Nov 03 22:12:18.090997 ip-10-0-128-81 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:12:18.490765 ip-10-0-128-81 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:18.490765 ip-10-0-128-81 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:12:18.490765 ip-10-0-128-81 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:18.490765 ip-10-0-128-81 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:18.495424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495344 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.495424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495423 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.495424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495427 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495431 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495434 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495437 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495440 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495442 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495445 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495447 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495450 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495453 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495455 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495458 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495461 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495464 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495466 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495469 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495473 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495478 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495481 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.495520 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495484 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495487 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495503 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495507 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495510 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495513 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495517 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495519 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495522 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495525 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495528 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495530 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495533 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495536 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495539 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495541 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495544 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495546 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495550 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.496001 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495554 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495556 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495559 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495561 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495564 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495567 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495569 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495571 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495574 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495577 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495580 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495583 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495586 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495588 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495591 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495594 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495597 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495600 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495603 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495605 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.496521 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495608 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495611 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495613 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495616 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495618 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495621 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495623 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495626 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495628 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495631 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495633 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495636 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495638 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495641 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495644 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495646 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495649 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495651 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495655 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.497017 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495657 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495660 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495662 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495665 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495667 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495670 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495672 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495675 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495678 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495681 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495683 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495686 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495688 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495698 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495703 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495706 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495709 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495711 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495726 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495728 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.497481 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495731 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495733 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495736 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495739 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495741 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495744 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.495747 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496494 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496500 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496503 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496506 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496509 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496512 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496515 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496518 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496521 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496524 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496527 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496530 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496533 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.498009 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496536 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496539 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496541 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496546 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496550 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496552 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496556 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496558 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496561 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496563 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496566 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496569 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496571 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496573 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496576 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496578 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496583 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496587 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496590 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.498511 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496593 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496596 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496598 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496601 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496604 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496606 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496609 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496612 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496615 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496618 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496620 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496623 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496625 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496629 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496632 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496634 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496637 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496640 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496643 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496645 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.498978 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496648 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496650 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496653 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496655 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496657 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496660 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496662 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496665 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496667 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496670 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496672 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496675 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496678 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496680 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496683 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496685 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496687 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496690 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496692 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496695 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.499459 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496698 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496701 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496704 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496706 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496709 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496712 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496729 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496732 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496734 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496737 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496740 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496742 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496744 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496747 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496750 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496753 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496755 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496758 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496760 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496763 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.499945 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496765 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496768 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496771 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496773 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496776 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496779 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496782 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496785 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496788 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496790 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496793 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496796 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496799 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.496801 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497283 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497288 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497291 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497294 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497297 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497299 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.500424 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497302 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497305 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497308 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497311 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497313 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497316 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497319 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497321 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497324 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497331 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497335 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497338 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497342 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497345 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497348 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497351 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497353 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497356 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497359 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.500926 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497361 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497364 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497367 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497369 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497372 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497375 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497378 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497380 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497383 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497385 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497388 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497391 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497393 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497396 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497398 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497401 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497404 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497406 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497409 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497411 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.501376 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497414 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497417 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497421 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497424 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497426 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497429 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497431 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497434 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497437 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497439 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497442 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497445 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497447 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497450 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497453 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497455 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497458 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497460 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497463 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497466 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.501878 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497468 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497470 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497473 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497476 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497478 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497481 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497484 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497487 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497490 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497492 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497495 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497498 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497500 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497503 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497507 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497510 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497512 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497515 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497517 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497520 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.502375 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497522 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497525 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497527 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497530 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497532 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497535 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497537 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497540 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497542 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497545 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497547 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497556 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497558 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497561 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497566 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497569 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497572 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497574 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497577 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.502886 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497579 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.497582 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498591 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498601 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498607 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498612 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498616 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498620 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498624 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498628 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498631 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498635 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498639 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498642 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498645 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498648 2581 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498651 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498655 2581 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498657 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498660 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498665 2581 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498668 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498671 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498674 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:12:18.503367 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498677 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498681 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498684 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498689 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498693 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498696 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498699 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498702 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498705 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498708 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498722 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498726 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498728 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498731 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498735 2581 flags.go:64] FLAG: --enable-server="true" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498738 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498742 2581 flags.go:64] FLAG: --event-burst="100" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498745 2581 flags.go:64] FLAG: --event-qps="50" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498748 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498751 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498754 2581 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498759 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498762 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498765 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498768 2581 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:12:18.503933 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498772 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498775 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498778 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498781 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498784 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498787 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498790 2581 flags.go:64] FLAG: --feature-gates="" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498794 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498797 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498801 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498806 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498810 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498813 2581 flags.go:64] FLAG: --help="false" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498816 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498819 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498822 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498825 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498828 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498832 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498835 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498838 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498840 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498843 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:12:18.504570 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498846 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498850 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498853 2581 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498856 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498859 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498862 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498864 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498867 2581 flags.go:64] FLAG: --lock-file="" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498870 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498873 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498876 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498884 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498887 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498890 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498893 2581 flags.go:64] FLAG: --logging-format="text" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498896 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498900 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498902 2581 flags.go:64] FLAG: --manifest-url="" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498905 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498912 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498915 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498919 2581 flags.go:64] FLAG: --max-pods="110" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498922 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498925 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498928 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:12:18.505230 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498931 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498934 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498937 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498940 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498947 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498950 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498953 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498956 2581 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498959 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498962 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498965 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498968 2581 flags.go:64] FLAG: --port="10250" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498971 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498974 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de017e703e244506" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498977 2581 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498981 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498984 2581 flags.go:64] FLAG: --register-node="true" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498987 2581 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498991 2581 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498994 2581 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.498997 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499000 2581 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499004 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499007 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499010 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499012 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:12:18.505845 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499017 2581 flags.go:64] FLAG: --runonce="false" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499020 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499023 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499026 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499029 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499032 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499035 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499040 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499043 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499046 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499049 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499052 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499055 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499059 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499061 2581 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499064 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499070 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499073 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499075 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499080 2581 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499083 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499086 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499089 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499092 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499094 2581 flags.go:64] FLAG: --v="2" Nov 03 22:12:18.506472 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499099 2581 flags.go:64] FLAG: --version="false" Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499103 2581 flags.go:64] FLAG: --vmodule="" Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499107 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499110 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499248 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499252 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499255 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499259 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499262 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499265 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499268 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499270 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499273 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499275 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499279 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499282 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499285 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499288 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499290 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499293 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.507622 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499296 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499299 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499301 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499304 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499306 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499309 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499312 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499314 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499317 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499319 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499322 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499325 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499327 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499330 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499332 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499335 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499337 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499340 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499342 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499346 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.508127 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499349 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499352 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499356 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499359 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499361 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499364 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499368 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499370 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499373 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499376 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499378 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499381 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499384 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499386 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499389 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499392 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499394 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499397 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499400 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499403 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.508631 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499405 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499408 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499410 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499413 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499416 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499418 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499421 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499423 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499426 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499428 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499431 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499435 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499438 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499440 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499443 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499446 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499449 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499452 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499455 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499458 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.509181 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499461 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499464 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499467 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499470 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499472 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499475 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499478 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499480 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499483 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499485 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499488 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499492 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499496 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499499 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499502 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499505 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499508 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499510 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499513 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.509659 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499516 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499518 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499521 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499525 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499530 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499532 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499535 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499538 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499541 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499544 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499546 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.499575 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499609 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499614 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499618 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.510126 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499621 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499624 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499627 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499630 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499633 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499635 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499638 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499641 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499643 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499646 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499648 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499651 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499654 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499657 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499659 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499662 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499665 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499667 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499670 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.510505 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499672 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499675 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499679 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499682 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499684 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499687 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499689 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499692 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499695 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499698 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499702 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499705 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499707 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499710 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499723 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499726 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499729 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499732 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499735 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499737 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.510993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499740 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499743 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499745 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499748 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499750 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499753 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499756 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499759 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499762 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499764 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499767 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499769 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499772 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499775 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499779 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499781 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499788 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499791 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499793 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499796 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.511482 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499799 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499802 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499805 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499808 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499810 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499813 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499816 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499818 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499821 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499824 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499826 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499829 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499832 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499835 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499837 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499840 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499843 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499845 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499848 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499850 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.511993 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499853 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499856 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499858 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499861 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499866 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499870 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499872 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499875 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499879 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499882 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499886 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499889 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499892 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499896 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499899 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499902 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499905 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499908 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.512466 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499910 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499914 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499918 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499921 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.499925 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.507270 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.507285 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.507301 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.507306 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507361 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507368 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507371 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507374 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507377 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507380 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507382 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507385 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507388 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507390 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.513000 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507394 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507396 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507399 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507403 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507407 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507411 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507413 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507416 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507419 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507422 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507424 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507427 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507430 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507432 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507435 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507437 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507440 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507443 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507446 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.513499 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507449 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507452 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507455 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507457 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507460 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507462 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507465 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507467 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507470 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507472 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507475 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507477 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507480 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507483 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507486 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507488 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507491 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507493 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507496 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.514025 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507498 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507501 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507503 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507506 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507508 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507511 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507513 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507516 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507518 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507521 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507523 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507526 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507530 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507533 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507536 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507539 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507542 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507544 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507547 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.514526 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507550 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507552 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507555 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507558 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507560 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507563 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507565 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507568 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507571 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507573 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507576 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507578 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507581 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507583 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507586 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507588 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507591 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507594 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507596 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507599 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.514997 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507602 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507604 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507607 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507609 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507612 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507614 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507617 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507619 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507622 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507625 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507628 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507630 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507633 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507636 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507638 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507641 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507643 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507646 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.515796 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507648 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.507675 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507831 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507835 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507839 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507841 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507844 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507847 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507850 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507852 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507855 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507858 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507860 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507863 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507866 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507868 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.516395 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507871 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507874 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507876 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507878 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507881 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507884 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507886 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507889 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507892 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507894 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507897 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507900 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507903 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507905 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507908 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507910 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507913 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507915 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507918 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507920 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.517040 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507923 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507925 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507928 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507930 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507933 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507935 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507938 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507940 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507943 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507945 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507948 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507950 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507953 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507955 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507958 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507960 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507963 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507965 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507967 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507970 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.517549 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507973 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507975 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507979 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507982 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507985 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507988 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507991 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507993 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.507996 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508001 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508003 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508005 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508008 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508011 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508013 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508016 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508019 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508021 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508023 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.518091 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508026 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508028 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508031 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508034 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508036 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508039 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508041 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508043 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508046 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508049 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508051 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508053 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508056 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508058 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508061 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508064 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508067 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508069 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508072 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.518674 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508074 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508078 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508082 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508087 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508090 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508093 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508095 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508098 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508100 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508103 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508105 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508108 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508111 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508113 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.508136 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:18.519221 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508170 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508175 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508178 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508181 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508184 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508187 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508190 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508193 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508196 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508199 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508201 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508204 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508207 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508210 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508212 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508215 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508218 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508220 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508223 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508225 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:18.519572 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508230 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508233 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508235 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508238 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508240 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508243 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508245 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508248 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508251 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508253 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508256 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508258 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508261 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508263 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508266 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508268 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508271 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508273 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508276 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508278 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:18.520073 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508281 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508283 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508286 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508289 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508291 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508294 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508300 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508303 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508306 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508308 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508311 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508313 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508318 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508320 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508322 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508325 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508327 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508330 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508333 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:18.520539 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508336 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508339 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508342 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508344 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508347 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508350 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508353 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508355 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508358 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508360 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508363 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508366 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508368 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508371 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508374 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508378 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508380 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508383 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508386 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508389 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:18.521004 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508393 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508395 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508398 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508400 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508404 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508407 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508410 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508413 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508415 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508418 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508420 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508423 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508425 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508428 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508430 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508433 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508436 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508439 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508443 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:18.521483 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508446 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508449 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508452 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508455 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508457 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508460 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508463 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:18.508466 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.508534 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.510581 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.511622 2581 server.go:1014] "Starting client certificate rotation" Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.511731 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.511767 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.513656 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:18.521985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.519543 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:18.533095 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.533077 2581 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:12:18.536247 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.536230 2581 log.go:25] "Validated CRI v1 image API" Nov 03 22:12:18.537224 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.537198 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:12:18.544134 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.544110 2581 fs.go:135] Filesystem UUIDs: map[0e9f578c-82a3-451b-8bed-3986666ebf75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e3b71187-b4e3-4782-8520-87e0c365672c:/dev/nvme0n1p3] Nov 03 22:12:18.544181 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.544134 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:12:18.549351 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.549229 2581 manager.go:217] Machine: {Timestamp:2026-11-03 22:12:18.547951002 +0000 UTC m=+0.340079457 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100131 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0f8bf6504e2c0edc5b1275b762ed SystemUUID:ec2d0f8b-f650-4e2c-0edc-5b1275b762ed BootID:1db81baa-5719-4853-b841-3fb9639f739a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:ab:fd:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:ab:fd:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:41:9e:30:09:3f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:12:18.549351 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.549345 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:12:18.549470 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.549432 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:12:18.551173 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.551146 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:12:18.551319 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.551176 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:12:18.551364 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.551322 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:12:18.551364 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.551330 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:12:18.551414 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.551370 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:18.551414 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.551390 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:18.552118 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.552107 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:12:18.554688 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.554676 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:12:18.554820 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.554811 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:12:18.555485 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.555474 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:12:18.555518 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.555490 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:12:18.555518 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.555500 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:12:18.555518 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.555509 2581 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:12:18.555518 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.555518 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:12:18.555879 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.555868 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:18.558113 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.558097 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:12:18.562553 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.562538 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:18.563344 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563333 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:12:18.563380 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563363 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:12:18.563515 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563505 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:12:18.563548 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563521 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:12:18.563548 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563535 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:12:18.563548 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563541 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:12:18.563548 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563547 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:12:18.563668 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.563553 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:12:18.564024 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564016 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:12:18.564060 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564025 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:12:18.564060 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564032 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:12:18.564060 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564038 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:12:18.564060 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564050 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:12:18.564060 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564059 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:12:18.564703 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564693 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:12:18.564703 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.564703 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:12:18.568711 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.568696 2581 server.go:1263] "Started kubelet" Nov 03 22:12:18.569076 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.569058 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:12:18.569320 ip-10-0-128-81 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:12:18.570646 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.570570 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:12:18.570749 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.570692 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:12:18.574507 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.574484 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:12:18.574585 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.574570 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:12:18.579768 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.579747 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:12:18.580507 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.580486 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:12:18.580636 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.580618 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:18.581282 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581261 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:12:18.581397 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.581371 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:12:18.581458 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581415 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:12:18.581458 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581430 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:12:18.581526 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581503 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:12:18.581526 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581513 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:12:18.581591 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581536 2581 factory.go:55] Registering systemd factory Nov 03 22:12:18.581591 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581548 2581 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:12:18.581872 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581821 2581 factory.go:153] Registering CRI-O factory Nov 03 22:12:18.581872 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581838 2581 factory.go:223] Registration of the crio container factory successfully Nov 03 22:12:18.581971 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581881 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:12:18.581971 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581897 2581 factory.go:103] Registering Raw factory Nov 03 22:12:18.581971 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.581911 2581 manager.go:1199] Started watching for new ooms in manager Nov 03 22:12:18.582831 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.582815 2581 manager.go:319] Starting recovery of all containers Nov 03 22:12:18.593875 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.593860 2581 manager.go:324] Recovery completed Nov 03 22:12:18.598940 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.598923 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:18.601435 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.601415 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:18.601523 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.601453 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:18.601523 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.601467 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:18.602054 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.602041 2581 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:12:18.602113 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.602054 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:12:18.602113 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.602075 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:12:18.602113 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.602045 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:18.603140 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.603100 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-81.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:12:18.603218 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.603135 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:12:18.603267 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.603226 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:18.603267 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.603255 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:18.603320 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.603268 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:18.603320 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.603311 2581 policy_none.go:50] "Start" Nov 03 22:12:18.603369 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.603324 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:12:18.603369 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.603336 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:12:18.604013 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.603011 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-81.ec2.internal.18e4a89bc462f87d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-81.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-81.ec2.internal,},FirstTimestamp:2026-11-03 22:12:18.568517757 +0000 UTC m=+0.360646211,LastTimestamp:2026-11-03 22:12:18.568517757 +0000 UTC m=+0.360646211,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-81.ec2.internal,}" Nov 03 22:12:18.604420 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.604398 2581 policy_none.go:44] "Start" Nov 03 22:12:18.613926 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.613907 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:12:18.613985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.613946 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nlf4h" Nov 03 22:12:18.622216 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.622195 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nlf4h" Nov 03 22:12:18.628469 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.628393 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-81.ec2.internal.18e4a89bc6593ce8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-81.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-81.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-81.ec2.internal,},FirstTimestamp:2026-11-03 22:12:18.601434344 +0000 UTC m=+0.393562801,LastTimestamp:2026-11-03 22:12:18.601434344 +0000 UTC m=+0.393562801,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-81.ec2.internal,}" Nov 03 22:12:18.628564 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.628484 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 22:12:18.639349 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.639320 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:12:18.640571 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.640542 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:12:18.640642 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.640568 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 22:12:18.643634 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.643619 2581 manager.go:341] "Starting Device Plugin manager" Nov 03 22:12:18.643743 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.643650 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:12:18.643743 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.643661 2581 server.go:87] "Starting device plugin registration server" Nov 03 22:12:18.643938 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.643925 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:12:18.644001 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.643941 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:12:18.644051 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.644033 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:12:18.644116 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.644103 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:12:18.644116 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.644116 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:12:18.644550 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.644522 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:12:18.644636 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.644578 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:18.729890 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.729860 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:12:18.731152 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.731135 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:12:18.731152 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.731155 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:12:18.731315 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.731175 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:12:18.731315 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.731207 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 22:12:18.744147 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.743974 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:18.744147 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.744140 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:18.745577 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.745558 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:18.745677 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.745598 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:18.745677 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.745612 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:18.745677 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.745638 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.763435 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.763419 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.763502 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.763443 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-81.ec2.internal\": node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:18.795901 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.795858 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:18.831857 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.831823 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"] Nov 03 22:12:18.831945 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.831895 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:18.833004 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.832977 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:18.833135 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.833022 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:18.833135 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.833035 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:18.834281 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.834269 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:18.834430 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.834414 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.834469 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.834446 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:18.835129 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.835112 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:18.835129 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.835120 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:18.835265 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.835153 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:18.835265 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.835165 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:18.835265 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.835154 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:18.835265 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.835205 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:18.836366 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.836352 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.836410 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.836385 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:18.837131 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.837116 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:18.837192 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.837146 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:18.837192 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.837156 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:18.861220 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.861201 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-81.ec2.internal\" not found" node="ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.864847 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.864830 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-81.ec2.internal\" not found" node="ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.882815 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.882793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.882901 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.882817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbffddcf197bf46e398ebd8b03a5cd7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"6cbffddcf197bf46e398ebd8b03a5cd7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.882901 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.882834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.896898 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.896882 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:18.983378 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.983345 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.983524 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.983391 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbffddcf197bf46e398ebd8b03a5cd7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"6cbffddcf197bf46e398ebd8b03a5cd7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.983524 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.983407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.983524 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.983430 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.983524 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.983448 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbffddcf197bf46e398ebd8b03a5cd7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"6cbffddcf197bf46e398ebd8b03a5cd7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.983524 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:18.983441 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:18.997019 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:18.996976 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.097648 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.097618 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.164848 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.164811 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:19.168604 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.168583 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:19.198282 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.198238 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.298700 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.298674 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.399095 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.399070 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.499527 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.499502 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.511644 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.511627 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:12:19.511801 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.511780 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:12:19.600324 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.600290 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.625638 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.625596 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:07:18 +0000 UTC" deadline="2028-05-30 16:30:10.855380644 +0000 UTC" Nov 03 22:12:19.625638 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.625624 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13770h17m51.229760131s" Nov 03 22:12:19.638252 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:19.638217 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c11f74aa32ce1bf6de4a6284c11142.slice/crio-dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039 WatchSource:0}: Error finding container dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039: Status 404 returned error can't find the container with id dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039 Nov 03 22:12:19.642296 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:19.642272 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbffddcf197bf46e398ebd8b03a5cd7.slice/crio-864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671 WatchSource:0}: Error finding container 864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671: Status 404 returned error can't find the container with id 864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671 Nov 03 22:12:19.643589 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.643569 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:12:19.653976 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.653960 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:19.654846 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.654831 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:19.654904 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.654860 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:19.654904 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.654869 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:19.700521 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.700493 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.732939 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.732909 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:19.734683 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.734642 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"03c11f74aa32ce1bf6de4a6284c11142","Type":"ContainerStarted","Data":"dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039"} Nov 03 22:12:19.735662 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:19.735645 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" event={"ID":"6cbffddcf197bf46e398ebd8b03a5cd7","Type":"ContainerStarted","Data":"864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671"} Nov 03 22:12:19.801137 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.801113 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:19.901618 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:19.901536 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:20.002004 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.001971 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:20.075220 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.075190 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:20.081521 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.081497 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:20.102767 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.102746 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:20.104698 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.104677 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:20.126168 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.126140 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:20.200001 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.199928 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:20.556055 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.555979 2581 apiserver.go:52] "Watching apiserver" Nov 03 22:12:20.580438 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.580413 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:12:20.585170 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.585148 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:12:20.587592 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.587059 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal","openshift-multus/multus-additional-cni-plugins-qphlr","openshift-ovn-kubernetes/ovnkube-node-49k6b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6","openshift-multus/multus-4t752","openshift-multus/network-metrics-daemon-pwbkp","openshift-network-diagnostics/network-check-target-xpshj","openshift-network-operator/iptables-alerter-f9gq2","openshift-dns/node-resolver-97blh","openshift-image-registry/node-ca-g7nfl","kube-system/konnectivity-agent-qwdnc","kube-system/global-pull-secret-syncer-nvbcj","kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal","openshift-cluster-node-tuning-operator/tuned-c26j6"] Nov 03 22:12:20.588798 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.588777 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:20.588913 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.588871 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:20.589763 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.589746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.591278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.591257 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.592463 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.592445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.593981 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.593964 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4t752" Nov 03 22:12:20.595429 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.595409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:20.595505 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.595479 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:20.596525 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.596507 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.597922 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.597903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.599054 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.599038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.600907 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.600885 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.602392 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.602060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.603298 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.603277 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.603381 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.603346 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:20.605453 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.605412 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:12:20.605563 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.605480 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4jdd\"" Nov 03 22:12:20.605563 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.605514 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:12:20.606788 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.606771 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:12:20.608091 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608069 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:12:20.608310 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608292 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:12:20.608416 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608401 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:12:20.608486 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608469 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:12:20.608799 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608781 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:12:20.608883 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608855 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz46p\"" Nov 03 22:12:20.608940 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608927 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wf9zk\"" Nov 03 22:12:20.608986 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.608952 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:12:20.609032 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.609009 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:12:20.609105 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.609076 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:12:20.609147 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.609137 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:12:20.612185 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.612045 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:12:20.612185 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.612108 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:12:20.612185 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.612147 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nt8s4\"" Nov 03 22:12:20.612185 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.612153 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:12:20.612185 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.612164 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:12:20.612643 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.612629 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwdqh\"" Nov 03 22:12:20.616293 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.616278 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:12:20.620866 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.620780 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:20.620866 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.620829 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wvrhf\"" Nov 03 22:12:20.620866 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.620865 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:20.620969 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.620943 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:12:20.621010 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.620997 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:12:20.621122 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621094 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:12:20.621170 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621106 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdchn\"" Nov 03 22:12:20.621204 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621170 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:20.621262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621246 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:12:20.621262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621247 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7shc8\"" Nov 03 22:12:20.621490 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621472 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:12:20.621554 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621522 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:20.621706 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621688 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tpvw\"" Nov 03 22:12:20.621809 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.621782 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:12:20.636571 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.636547 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:12:20.636676 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.636623 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kg8gp" Nov 03 22:12:20.682479 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.682454 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:12:20.693821 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.693791 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-run\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.693977 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.693833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jtl\" (UniqueName: \"kubernetes.io/projected/f0705c0f-2c37-412d-a593-76380e0c37e0-kube-api-access-m8jtl\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.693977 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.693945 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.694086 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.693981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-slash\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.694086 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lnxp\" (UniqueName: \"kubernetes.io/projected/6294b9cc-3007-4941-bff4-0300e2f6343c-kube-api-access-5lnxp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.694086 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:20.694086 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694054 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.694086 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694077 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-sys\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-netns\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-kubelet\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-multus-certs\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2165f0f-049f-4588-898f-db7ef275411d-konnectivity-ca\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wnh\" (UniqueName: \"kubernetes.io/projected/05f6162d-1577-465d-83c3-b020607dd4e9-kube-api-access-t2wnh\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-lib-modules\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvs5\" (UniqueName: \"kubernetes.io/projected/941fe33c-1af4-4a8c-bbda-6c3756edae7f-kube-api-access-rjvs5\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-dbus\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694225 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-systemd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-bin\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.694278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-modprobe-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-sys-fs\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cni-binary-copy\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694330 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-conf-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4fdl\" (UniqueName: \"kubernetes.io/projected/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-kube-api-access-j4fdl\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-ovn\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0705c0f-2c37-412d-a593-76380e0c37e0-host-slash\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-daemon-config\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-etc-kubernetes\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694479 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694506 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkqf\" (UniqueName: \"kubernetes.io/projected/52a290a0-8456-4ca6-afa3-8324f7c76cc0-kube-api-access-jqkqf\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694538 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-kubelet-config\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-hostroot\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hh6m\" (UniqueName: \"kubernetes.io/projected/a04401b8-ddb6-4fed-bac2-2526d011a0ad-kube-api-access-5hh6m\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694621 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-systemd-units\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.694702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-etc-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-socket-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-device-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-bin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694765 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-hosts-file\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-kube-api-access-ntqgq\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-kubernetes\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694876 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-var-lib-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-netd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694922 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694946 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-system-cni-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.694990 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-host\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-serviceca\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.695365 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-registration-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-kubelet\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695106 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695130 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-system-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-k8s-cni-cncf-io\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-multus\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-netns\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-script-lib\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-conf\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-tuned\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695298 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5w6l\" (UniqueName: \"kubernetes.io/projected/e55cd659-eeee-4fbf-b11b-c376d306a652-kube-api-access-s5w6l\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695321 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-log-socket\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cnibin\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0705c0f-2c37-412d-a593-76380e0c37e0-iptables-alerter-script\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.695855 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-node-log\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-config\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695471 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysconfig\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-systemd\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2165f0f-049f-4588-898f-db7ef275411d-agent-certs\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-env-overrides\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695590 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695614 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-tmp-dir\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695644 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-var-lib-kubelet\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-host\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695691 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-tmp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cnibin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-socket-dir-parent\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.696262 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.695774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55cd659-eeee-4fbf-b11b-c376d306a652-ovn-node-metrics-cert\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.796929 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.796903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-multus\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.797030 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.796985 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-multus\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.797096 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-netns\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797096 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-script-lib\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797183 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-conf\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.797183 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797099 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-netns\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797183 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797118 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-tuned\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.797183 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5w6l\" (UniqueName: \"kubernetes.io/projected/e55cd659-eeee-4fbf-b11b-c376d306a652-kube-api-access-s5w6l\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797346 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797346 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797261 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-conf\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.797346 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-log-socket\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797346 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797311 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cnibin\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.797346 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797346 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0705c0f-2c37-412d-a593-76380e0c37e0-iptables-alerter-script\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.797346 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797348 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cnibin\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-node-log\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797378 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-log-socket\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797382 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-config\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797402 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-node-log\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysconfig\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-systemd\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2165f0f-049f-4588-898f-db7ef275411d-agent-certs\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797448 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797527 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-systemd\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797579 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysconfig\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-env-overrides\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797646 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-tmp-dir\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.797702 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-var-lib-kubelet\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797730 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-script-lib\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-host\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-tmp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cnibin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-socket-dir-parent\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55cd659-eeee-4fbf-b11b-c376d306a652-ovn-node-metrics-cert\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797854 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-run\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797879 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jtl\" (UniqueName: \"kubernetes.io/projected/f0705c0f-2c37-412d-a593-76380e0c37e0-kube-api-access-m8jtl\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-slash\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lnxp\" (UniqueName: \"kubernetes.io/projected/6294b9cc-3007-4941-bff4-0300e2f6343c-kube-api-access-5lnxp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797956 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-config\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0705c0f-2c37-412d-a593-76380e0c37e0-iptables-alerter-script\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797974 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.797997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798018 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-tmp-dir\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-sys\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.798341 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-netns\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798059 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-host\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.798064 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798061 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-sys\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-kubelet\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798090 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-slash\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798101 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-kubelet\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-multus-certs\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.798121 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret podName:17608ecb-fd95-49bf-a939-b2c1587e4fa1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:21.298104703 +0000 UTC m=+3.090233157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret") pod "global-pull-secret-syncer-nvbcj" (UID: "17608ecb-fd95-49bf-a939-b2c1587e4fa1") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798227 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-run\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798395 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-env-overrides\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798021 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-var-lib-kubelet\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798450 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798460 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-netns\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2165f0f-049f-4588-898f-db7ef275411d-konnectivity-ca\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798504 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-socket-dir-parent\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798540 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cnibin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wnh\" (UniqueName: \"kubernetes.io/projected/05f6162d-1577-465d-83c3-b020607dd4e9-kube-api-access-t2wnh\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.799119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-lib-modules\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvs5\" (UniqueName: \"kubernetes.io/projected/941fe33c-1af4-4a8c-bbda-6c3756edae7f-kube-api-access-rjvs5\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798624 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-dbus\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-systemd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-bin\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798760 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-dbus\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798790 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-modprobe-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-sys-fs\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cni-binary-copy\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-conf-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4fdl\" (UniqueName: \"kubernetes.io/projected/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-kube-api-access-j4fdl\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-ovn\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0705c0f-2c37-412d-a593-76380e0c37e0-host-slash\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798956 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-daemon-config\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.798977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-etc-kubernetes\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799010 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-lib-modules\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.799927 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkqf\" (UniqueName: \"kubernetes.io/projected/52a290a0-8456-4ca6-afa3-8324f7c76cc0-kube-api-access-jqkqf\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.799077 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799117 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-kubelet-config\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.799133 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs podName:a04401b8-ddb6-4fed-bac2-2526d011a0ad nodeName:}" failed. No retries permitted until 2026-11-03 22:12:21.299115425 +0000 UTC m=+3.091243864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs") pod "network-metrics-daemon-pwbkp" (UID: "a04401b8-ddb6-4fed-bac2-2526d011a0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-kubelet-config\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799169 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-hostroot\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hh6m\" (UniqueName: \"kubernetes.io/projected/a04401b8-ddb6-4fed-bac2-2526d011a0ad-kube-api-access-5hh6m\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-systemd-units\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-etc-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799009 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2165f0f-049f-4588-898f-db7ef275411d-konnectivity-ca\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-socket-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799320 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-device-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-bin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-hosts-file\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.800611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-kube-api-access-ntqgq\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-kubernetes\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799467 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799501 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-conf-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799520 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-hostroot\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799544 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-systemd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799581 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-bin\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799589 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799655 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799661 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-modprobe-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799773 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-systemd-units\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-etc-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799878 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-ovn\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799916 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0705c0f-2c37-412d-a593-76380e0c37e0-host-slash\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-socket-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799949 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-device-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799989 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-bin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.801358 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800073 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-hosts-file\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.799468 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-var-lib-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-netd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-sys-fs\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800205 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-etc-kubernetes\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800307 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-var-lib-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-system-cni-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800286 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-kubernetes\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-host\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800356 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-serviceca\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800316 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-netd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-registration-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800408 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-host\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.802137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800419 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-kubelet\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-system-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800494 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-registration-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800358 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-k8s-cni-cncf-io\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800542 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-kubelet\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-system-cni-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800577 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-k8s-cni-cncf-io\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800649 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cni-binary-copy\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800741 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-system-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.800899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-serviceca\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.801006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.801014 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-multus-certs\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.801230 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-daemon-config\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.801405 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.802823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.801492 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-tuned\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.803450 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.801756 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2165f0f-049f-4588-898f-db7ef275411d-agent-certs\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.803450 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.802223 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55cd659-eeee-4fbf-b11b-c376d306a652-ovn-node-metrics-cert\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.803450 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.802975 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-tmp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.818336 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.818289 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:20.818498 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.818484 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:20.818607 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.818590 2581 projected.go:196] Error preparing data for projected volume kube-api-access-p6gdd for pod openshift-network-diagnostics/network-check-target-xpshj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:20.818740 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:20.818668 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd podName:a6ad98a8-5e26-471c-a5a0-0ea41c026e1c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:21.318645469 +0000 UTC m=+3.110773928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p6gdd" (UniqueName: "kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd") pod "network-check-target-xpshj" (UID: "a6ad98a8-5e26-471c-a5a0-0ea41c026e1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:20.820683 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.820661 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lnxp\" (UniqueName: \"kubernetes.io/projected/6294b9cc-3007-4941-bff4-0300e2f6343c-kube-api-access-5lnxp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.820783 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.820751 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jtl\" (UniqueName: \"kubernetes.io/projected/f0705c0f-2c37-412d-a593-76380e0c37e0-kube-api-access-m8jtl\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.821504 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.821477 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5w6l\" (UniqueName: \"kubernetes.io/projected/e55cd659-eeee-4fbf-b11b-c376d306a652-kube-api-access-s5w6l\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.822639 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.822613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wnh\" (UniqueName: \"kubernetes.io/projected/05f6162d-1577-465d-83c3-b020607dd4e9-kube-api-access-t2wnh\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.822868 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.822852 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkqf\" (UniqueName: \"kubernetes.io/projected/52a290a0-8456-4ca6-afa3-8324f7c76cc0-kube-api-access-jqkqf\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.823564 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.823548 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvs5\" (UniqueName: \"kubernetes.io/projected/941fe33c-1af4-4a8c-bbda-6c3756edae7f-kube-api-access-rjvs5\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:20.823649 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.823570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4fdl\" (UniqueName: \"kubernetes.io/projected/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-kube-api-access-j4fdl\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.825139 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.825118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-kube-api-access-ntqgq\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.825553 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.825537 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hh6m\" (UniqueName: \"kubernetes.io/projected/a04401b8-ddb6-4fed-bac2-2526d011a0ad-kube-api-access-5hh6m\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:20.903041 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.903012 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:20.911037 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:20.911011 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2165f0f_049f_4588_898f_db7ef275411d.slice/crio-bff27fde53144f474e8e4ca5cfe745e5eb2a85a6444c7123a61c1c595d4d6c50 WatchSource:0}: Error finding container bff27fde53144f474e8e4ca5cfe745e5eb2a85a6444c7123a61c1c595d4d6c50: Status 404 returned error can't find the container with id bff27fde53144f474e8e4ca5cfe745e5eb2a85a6444c7123a61c1c595d4d6c50 Nov 03 22:12:20.920889 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.920858 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:20.925104 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.925087 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:20.926137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.926121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4t752" Nov 03 22:12:20.929190 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:20.929167 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55cd659_eeee_4fbf_b11b_c376d306a652.slice/crio-e2630804f37348b7e6d930f4b96f4fdd9d685d00c5f06484f901af01e0cc5015 WatchSource:0}: Error finding container e2630804f37348b7e6d930f4b96f4fdd9d685d00c5f06484f901af01e0cc5015: Status 404 returned error can't find the container with id e2630804f37348b7e6d930f4b96f4fdd9d685d00c5f06484f901af01e0cc5015 Nov 03 22:12:20.934790 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.932347 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:20.934790 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.932907 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:20.936246 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:20.936177 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941fe33c_1af4_4a8c_bbda_6c3756edae7f.slice/crio-3f3e4e964af0a63479dfd41c9f4c4e350dc6e1a862b216a6b1a3555eaa860949 WatchSource:0}: Error finding container 3f3e4e964af0a63479dfd41c9f4c4e350dc6e1a862b216a6b1a3555eaa860949: Status 404 returned error can't find the container with id 3f3e4e964af0a63479dfd41c9f4c4e350dc6e1a862b216a6b1a3555eaa860949 Nov 03 22:12:20.939310 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.939291 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:20.947753 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:20.947659 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f6162d_1577_465d_83c3_b020607dd4e9.slice/crio-548691e3d13dbd44053a2a8716d9dbe685fa1eafa2b3435c26241af81eba8b5a WatchSource:0}: Error finding container 548691e3d13dbd44053a2a8716d9dbe685fa1eafa2b3435c26241af81eba8b5a: Status 404 returned error can't find the container with id 548691e3d13dbd44053a2a8716d9dbe685fa1eafa2b3435c26241af81eba8b5a Nov 03 22:12:20.948095 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:20.948073 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6294b9cc_3007_4941_bff4_0300e2f6343c.slice/crio-cdf0cb7af07350e78ec20a66f988d495ccf6ddf93822a0ce833fa9877e111d8c WatchSource:0}: Error finding container cdf0cb7af07350e78ec20a66f988d495ccf6ddf93822a0ce833fa9877e111d8c: Status 404 returned error can't find the container with id cdf0cb7af07350e78ec20a66f988d495ccf6ddf93822a0ce833fa9877e111d8c Nov 03 22:12:20.948773 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:20.948755 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ada7c0_b6ab_4ba2_bc6d_990b007311a1.slice/crio-5fb1828bfc38c813678aea328f233a195d7ecf8c1ea18ee298404d8a595d604d WatchSource:0}: Error finding container 5fb1828bfc38c813678aea328f233a195d7ecf8c1ea18ee298404d8a595d604d: Status 404 returned error can't find the container with id 5fb1828bfc38c813678aea328f233a195d7ecf8c1ea18ee298404d8a595d604d Nov 03 22:12:20.955832 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.955815 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:20.957189 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.957173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:20.962249 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:20.962232 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:20.972420 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:20.972399 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0705c0f_2c37_412d_a593_76380e0c37e0.slice/crio-7e495d17d943b9577b3777b0f9b4282d657d02b3848e28f99fa5b6673e238741 WatchSource:0}: Error finding container 7e495d17d943b9577b3777b0f9b4282d657d02b3848e28f99fa5b6673e238741: Status 404 returned error can't find the container with id 7e495d17d943b9577b3777b0f9b4282d657d02b3848e28f99fa5b6673e238741 Nov 03 22:12:21.303404 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.303104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:21.303404 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.303360 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:21.303404 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.303391 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:21.303664 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.303424 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs podName:a04401b8-ddb6-4fed-bac2-2526d011a0ad nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.303406983 +0000 UTC m=+4.095535444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs") pod "network-metrics-daemon-pwbkp" (UID: "a04401b8-ddb6-4fed-bac2-2526d011a0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:21.303664 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.303472 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:21.303664 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.303506 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret podName:17608ecb-fd95-49bf-a939-b2c1587e4fa1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.303496707 +0000 UTC m=+4.095625149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret") pod "global-pull-secret-syncer-nvbcj" (UID: "17608ecb-fd95-49bf-a939-b2c1587e4fa1") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:21.404513 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.404477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:21.404685 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.404666 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:21.404780 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.404692 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:21.404780 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.404706 2581 projected.go:196] Error preparing data for projected volume kube-api-access-p6gdd for pod openshift-network-diagnostics/network-check-target-xpshj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:21.404780 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.404779 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd podName:a6ad98a8-5e26-471c-a5a0-0ea41c026e1c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.4047601 +0000 UTC m=+4.196888557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6gdd" (UniqueName: "kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd") pod "network-check-target-xpshj" (UID: "a6ad98a8-5e26-471c-a5a0-0ea41c026e1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:21.638462 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.638352 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:20 +0000 UTC" deadline="2028-04-17 15:01:11.069302687 +0000 UTC" Nov 03 22:12:21.638462 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.638415 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12736h48m49.430891862s" Nov 03 22:12:21.732912 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.732375 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:21.732912 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:21.732512 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:21.744415 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.744371 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwdnc" event={"ID":"c2165f0f-049f-4588-898f-db7ef275411d","Type":"ContainerStarted","Data":"bff27fde53144f474e8e4ca5cfe745e5eb2a85a6444c7123a61c1c595d4d6c50"} Nov 03 22:12:21.757788 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.757756 2581 generic.go:356] "Generic (PLEG): container finished" podID="03c11f74aa32ce1bf6de4a6284c11142" containerID="69627fcb47055014ff62d390b1e64624f05e7b248845faf3f00f6fb54b0b83fd" exitCode=0 Nov 03 22:12:21.757932 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.757842 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"03c11f74aa32ce1bf6de4a6284c11142","Type":"ContainerDied","Data":"69627fcb47055014ff62d390b1e64624f05e7b248845faf3f00f6fb54b0b83fd"} Nov 03 22:12:21.768415 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.768344 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f9gq2" event={"ID":"f0705c0f-2c37-412d-a593-76380e0c37e0","Type":"ContainerStarted","Data":"7e495d17d943b9577b3777b0f9b4282d657d02b3848e28f99fa5b6673e238741"} Nov 03 22:12:21.794676 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.794640 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerStarted","Data":"00c1e219b17813ca93a7a52190c2fd54cc03e406f50c62db1f35c64096b29a44"} Nov 03 22:12:21.805122 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.805024 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4t752" event={"ID":"941fe33c-1af4-4a8c-bbda-6c3756edae7f","Type":"ContainerStarted","Data":"3f3e4e964af0a63479dfd41c9f4c4e350dc6e1a862b216a6b1a3555eaa860949"} Nov 03 22:12:21.807994 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.807897 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"e2630804f37348b7e6d930f4b96f4fdd9d685d00c5f06484f901af01e0cc5015"} Nov 03 22:12:21.811975 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.811947 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" event={"ID":"6cbffddcf197bf46e398ebd8b03a5cd7","Type":"ContainerStarted","Data":"1aac7f42d8c95710572834b11351b688c6137c7777346eba154cfd9f9808e711"} Nov 03 22:12:21.861306 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.861146 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g7nfl" event={"ID":"e5eb73d7-8dd5-4e38-b312-9fd9f4881762","Type":"ContainerStarted","Data":"0eebdb82a457f55b8ffbf3a093e6ebf1efdb444bfcccb2df6f04a8b88c0461a7"} Nov 03 22:12:21.865019 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.864984 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c26j6" event={"ID":"6294b9cc-3007-4941-bff4-0300e2f6343c","Type":"ContainerStarted","Data":"cdf0cb7af07350e78ec20a66f988d495ccf6ddf93822a0ce833fa9877e111d8c"} Nov 03 22:12:21.869971 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.869942 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97blh" event={"ID":"63ada7c0-b6ab-4ba2-bc6d-990b007311a1","Type":"ContainerStarted","Data":"5fb1828bfc38c813678aea328f233a195d7ecf8c1ea18ee298404d8a595d604d"} Nov 03 22:12:21.878860 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:21.878834 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" event={"ID":"05f6162d-1577-465d-83c3-b020607dd4e9","Type":"ContainerStarted","Data":"548691e3d13dbd44053a2a8716d9dbe685fa1eafa2b3435c26241af81eba8b5a"} Nov 03 22:12:22.311748 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.311655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:22.311748 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.311711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:22.311960 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.311884 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:22.311960 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.311947 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret podName:17608ecb-fd95-49bf-a939-b2c1587e4fa1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:24.311926979 +0000 UTC m=+6.104055435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret") pod "global-pull-secret-syncer-nvbcj" (UID: "17608ecb-fd95-49bf-a939-b2c1587e4fa1") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:22.312348 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.312322 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:22.312446 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.312390 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs podName:a04401b8-ddb6-4fed-bac2-2526d011a0ad nodeName:}" failed. No retries permitted until 2026-11-03 22:12:24.312367407 +0000 UTC m=+6.104495863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs") pod "network-metrics-daemon-pwbkp" (UID: "a04401b8-ddb6-4fed-bac2-2526d011a0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:22.412370 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.412337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:22.412562 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.412501 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:22.412562 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.412523 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:22.412562 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.412536 2581 projected.go:196] Error preparing data for projected volume kube-api-access-p6gdd for pod openshift-network-diagnostics/network-check-target-xpshj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:22.412709 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.412589 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd podName:a6ad98a8-5e26-471c-a5a0-0ea41c026e1c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:24.412571797 +0000 UTC m=+6.204700254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6gdd" (UniqueName: "kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd") pod "network-check-target-xpshj" (UID: "a6ad98a8-5e26-471c-a5a0-0ea41c026e1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:22.639236 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.639132 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:20 +0000 UTC" deadline="2028-05-11 08:14:01.66472773 +0000 UTC" Nov 03 22:12:22.639236 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.639175 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13306h1m39.025557157s" Nov 03 22:12:22.734926 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.734860 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:22.735095 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.734995 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:22.735095 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.735033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:22.735216 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:22.735113 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:22.891566 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.891490 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"03c11f74aa32ce1bf6de4a6284c11142","Type":"ContainerStarted","Data":"adff9d78183c06f8fcf4262ba722367caca8714f8e6f67eca8a7743c20041a8b"} Nov 03 22:12:22.934305 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.933415 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" podStartSLOduration=3.933397517 podStartE2EDuration="3.933397517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:21.853521334 +0000 UTC m=+3.645649798" watchObservedRunningTime="2026-11-03 22:12:22.933397517 +0000 UTC m=+4.725525984" Nov 03 22:12:22.934305 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:22.933597 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" podStartSLOduration=2.933591554 podStartE2EDuration="2.933591554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:22.932975218 +0000 UTC m=+4.725103683" watchObservedRunningTime="2026-11-03 22:12:22.933591554 +0000 UTC m=+4.725720019" Nov 03 22:12:23.731691 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:23.731613 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:23.732146 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:23.731771 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:23.898316 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:23.898281 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerStarted","Data":"d61cc0ffd9f8dc3f8ea35397ea6c61eca25ee72a32811de0dff3c693e110d877"} Nov 03 22:12:24.328003 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.327964 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:24.328190 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.328068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:24.328190 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.328186 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:24.328295 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.328246 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs podName:a04401b8-ddb6-4fed-bac2-2526d011a0ad nodeName:}" failed. No retries permitted until 2026-11-03 22:12:28.328228012 +0000 UTC m=+10.120356460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs") pod "network-metrics-daemon-pwbkp" (UID: "a04401b8-ddb6-4fed-bac2-2526d011a0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:24.328658 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.328639 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:24.328755 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.328692 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret podName:17608ecb-fd95-49bf-a939-b2c1587e4fa1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:28.328678633 +0000 UTC m=+10.120807080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret") pod "global-pull-secret-syncer-nvbcj" (UID: "17608ecb-fd95-49bf-a939-b2c1587e4fa1") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:24.429066 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.429026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:24.429263 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.429241 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:24.429323 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.429274 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:24.429323 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.429287 2581 projected.go:196] Error preparing data for projected volume kube-api-access-p6gdd for pod openshift-network-diagnostics/network-check-target-xpshj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:24.429427 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.429345 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd podName:a6ad98a8-5e26-471c-a5a0-0ea41c026e1c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:28.429327957 +0000 UTC m=+10.221456455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6gdd" (UniqueName: "kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd") pod "network-check-target-xpshj" (UID: "a6ad98a8-5e26-471c-a5a0-0ea41c026e1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:24.732839 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.732809 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:24.733306 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.732943 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:24.733306 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.733038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:24.733306 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:24.733188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:24.903063 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.902983 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwdnc" event={"ID":"c2165f0f-049f-4588-898f-db7ef275411d","Type":"ContainerStarted","Data":"5b2f40805ad75e552b5ca2714a3715dc72e263a8b200030a6a75dd3119999b42"} Nov 03 22:12:24.905789 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.905760 2581 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="d61cc0ffd9f8dc3f8ea35397ea6c61eca25ee72a32811de0dff3c693e110d877" exitCode=0 Nov 03 22:12:24.905926 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.905838 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerDied","Data":"d61cc0ffd9f8dc3f8ea35397ea6c61eca25ee72a32811de0dff3c693e110d877"} Nov 03 22:12:24.994679 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:24.994620 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qwdnc" podStartSLOduration=3.471849886 podStartE2EDuration="6.994601738s" totalImagesPullingTime="3.522751852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:24.945770077 +0000 UTC m=+6.737898542" watchObservedRunningTime="2026-11-03 22:12:24.994601738 +0000 UTC m=+6.786730207" Nov 03 22:12:25.732473 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:25.731946 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:25.732473 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:25.732082 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:25.921788 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:25.921085 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g7nfl" event={"ID":"e5eb73d7-8dd5-4e38-b312-9fd9f4881762","Type":"ContainerStarted","Data":"2b174bc097f023422ec12bf163fe5432bc6adba000f715bf7eda78bbe73b5979"} Nov 03 22:12:25.963226 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:25.962335 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g7nfl" podStartSLOduration=3.284343423 podStartE2EDuration="7.962318187s" totalImagesPullingTime="4.677974764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:25.961961029 +0000 UTC m=+7.754089494" watchObservedRunningTime="2026-11-03 22:12:25.962318187 +0000 UTC m=+7.754446652" Nov 03 22:12:26.733494 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:26.732838 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:26.733494 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:26.733035 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:26.733494 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:26.733378 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:26.733494 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:26.733454 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:26.930106 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:26.930067 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" event={"ID":"05f6162d-1577-465d-83c3-b020607dd4e9","Type":"ContainerStarted","Data":"4b7f0b11382fd8cfe552cbf03b196ddff16b08e05011120397f77e701d30a47c"} Nov 03 22:12:27.732135 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:27.731489 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:27.732135 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:27.731628 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:27.766135 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:27.766109 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:12:27.941430 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:27.941359 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" event={"ID":"05f6162d-1577-465d-83c3-b020607dd4e9","Type":"ContainerStarted","Data":"251d760cf6373638cbac84e673648491d980d3b114f9545da6d07a79157b84ed"} Nov 03 22:12:28.366006 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.365105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:28.366006 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.365172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:28.366006 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.365349 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:28.366006 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.365421 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret podName:17608ecb-fd95-49bf-a939-b2c1587e4fa1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:36.365401345 +0000 UTC m=+18.157529791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret") pod "global-pull-secret-syncer-nvbcj" (UID: "17608ecb-fd95-49bf-a939-b2c1587e4fa1") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:28.366006 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.365890 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:28.366006 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.365955 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs podName:a04401b8-ddb6-4fed-bac2-2526d011a0ad nodeName:}" failed. No retries permitted until 2026-11-03 22:12:36.365938227 +0000 UTC m=+18.158066758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs") pod "network-metrics-daemon-pwbkp" (UID: "a04401b8-ddb6-4fed-bac2-2526d011a0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:28.466567 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.466489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:28.466744 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.466653 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:28.466744 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.466691 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:28.466744 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.466704 2581 projected.go:196] Error preparing data for projected volume kube-api-access-p6gdd for pod openshift-network-diagnostics/network-check-target-xpshj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:28.466939 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.466796 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd podName:a6ad98a8-5e26-471c-a5a0-0ea41c026e1c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:36.46677629 +0000 UTC m=+18.258904749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6gdd" (UniqueName: "kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd") pod "network-check-target-xpshj" (UID: "a6ad98a8-5e26-471c-a5a0-0ea41c026e1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:28.649159 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.649082 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:12:27.766132065Z","UUID":"0262de51-a0a6-4a3c-8aea-b3522bbdbed8","Handler":null,"Name":"","Endpoint":""} Nov 03 22:12:28.650881 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.650862 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:12:28.650965 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.650901 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:12:28.733538 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.733473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:28.733660 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.733619 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:28.733985 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.733961 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:28.734094 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:28.734056 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:28.736463 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.736442 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:28.737141 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.737119 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:28.946807 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.946600 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97blh" event={"ID":"63ada7c0-b6ab-4ba2-bc6d-990b007311a1","Type":"ContainerStarted","Data":"1d06642ba6dca9debc68ee8a0277e5f0c84b98f1fe4e446d3c79cfd66efd8f1c"} Nov 03 22:12:28.950898 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.950873 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" event={"ID":"05f6162d-1577-465d-83c3-b020607dd4e9","Type":"ContainerStarted","Data":"480d0734f9372a828009117617e1ee52eda48586c2943e6be5aeaff8bada1ad3"} Nov 03 22:12:28.952581 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:28.952557 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4t752" event={"ID":"941fe33c-1af4-4a8c-bbda-6c3756edae7f","Type":"ContainerStarted","Data":"6a5435aedd126c0e52e35b953fcc74d7b0a3ca3f7e1bcd9d2d8e6de83c203e2c"} Nov 03 22:12:29.037990 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.037907 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" podStartSLOduration=3.499261582 podStartE2EDuration="11.037892591s" totalImagesPullingTime="7.538631009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:29.037612787 +0000 UTC m=+10.829741251" watchObservedRunningTime="2026-11-03 22:12:29.037892591 +0000 UTC m=+10.830021055" Nov 03 22:12:29.038247 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.038223 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-97blh" podStartSLOduration=3.476373971 podStartE2EDuration="11.038212074s" totalImagesPullingTime="7.561838103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:28.993212859 +0000 UTC m=+10.785341324" watchObservedRunningTime="2026-11-03 22:12:29.038212074 +0000 UTC m=+10.830340541" Nov 03 22:12:29.082093 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.082038 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4t752" podStartSLOduration=3.847730089 podStartE2EDuration="11.08201993s" totalImagesPullingTime="7.234289841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:29.081987566 +0000 UTC m=+10.874116031" watchObservedRunningTime="2026-11-03 22:12:29.08201993 +0000 UTC m=+10.874148392" Nov 03 22:12:29.731748 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.731701 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:29.731927 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:29.731830 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:29.956617 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.956578 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f9gq2" event={"ID":"f0705c0f-2c37-412d-a593-76380e0c37e0","Type":"ContainerStarted","Data":"78c17d93499007c5a3565eb0d6b74edd1aac71d81a6ba7241791b83f99ab2f1a"} Nov 03 22:12:29.958683 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.958651 2581 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="763702e8af47c24996acd88e1680903f78eaa74a9bb19a9d703125024c2048b7" exitCode=0 Nov 03 22:12:29.959639 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.959606 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerDied","Data":"763702e8af47c24996acd88e1680903f78eaa74a9bb19a9d703125024c2048b7"} Nov 03 22:12:29.999088 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:29.998994 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f9gq2" podStartSLOduration=4.792837554 podStartE2EDuration="11.998975879s" totalImagesPullingTime="7.206138325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:29.998008774 +0000 UTC m=+11.790137240" watchObservedRunningTime="2026-11-03 22:12:29.998975879 +0000 UTC m=+11.791104347" Nov 03 22:12:30.731577 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:30.731548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:30.731701 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:30.731594 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:30.731784 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:30.731700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:30.731912 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:30.731809 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:30.962750 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:30.962659 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c26j6" event={"ID":"6294b9cc-3007-4941-bff4-0300e2f6343c","Type":"ContainerStarted","Data":"a4158432ce53e53a04579f13c968d16217ac1ee6b16f210e63ddbb2cf838c83b"} Nov 03 22:12:30.965315 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:30.965282 2581 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="03dec4c5ce14c3f779f0e4deba10dabb990d92568078bafaf7117f51b5b06ed4" exitCode=0 Nov 03 22:12:30.965454 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:30.965360 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerDied","Data":"03dec4c5ce14c3f779f0e4deba10dabb990d92568078bafaf7117f51b5b06ed4"} Nov 03 22:12:31.009089 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:31.009040 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c26j6" podStartSLOduration=3.364354296 podStartE2EDuration="13.009028554s" totalImagesPullingTime="9.644674258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:31.008704516 +0000 UTC m=+12.800832981" watchObservedRunningTime="2026-11-03 22:12:31.009028554 +0000 UTC m=+12.801157018" Nov 03 22:12:31.731515 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:31.731488 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:31.731674 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:31.731601 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:31.970862 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:31.970818 2581 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="abe3e95b632faae57a1f5580a376c5f57b3e99366ca67d02719eee2b8fd5962f" exitCode=0 Nov 03 22:12:31.971423 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:31.970895 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerDied","Data":"abe3e95b632faae57a1f5580a376c5f57b3e99366ca67d02719eee2b8fd5962f"} Nov 03 22:12:32.732917 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:32.732380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:32.732917 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:32.732415 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:32.732917 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:32.732504 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:32.732917 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:32.732875 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:33.731899 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:33.731658 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:33.732261 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:33.732003 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:33.979138 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:33.979052 2581 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="c34023f39cf3c49e391a65beba1ef4da8801817ad7980d8ae957ee03252964ea" exitCode=0 Nov 03 22:12:33.979138 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:33.979102 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerDied","Data":"c34023f39cf3c49e391a65beba1ef4da8801817ad7980d8ae957ee03252964ea"} Nov 03 22:12:34.731639 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:34.731602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:34.731805 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:34.731733 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:34.731805 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:34.731777 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:34.731924 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:34.731902 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:34.984655 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:34.984577 2581 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="8e7b50840cfded42cd65bf5bc143fb965099fe3424f18e6f07b02a94f7a384f9" exitCode=0 Nov 03 22:12:34.984655 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:34.984618 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerDied","Data":"8e7b50840cfded42cd65bf5bc143fb965099fe3424f18e6f07b02a94f7a384f9"} Nov 03 22:12:35.731954 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.731711 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:35.732345 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:35.731988 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:35.989611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.989531 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"ddfbf8c95d5d4e1d236f27b6f42b77f3b9719a0b5bea7d0e85e482d3545f333b"} Nov 03 22:12:35.989611 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.989595 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"00f6d595a25d89b357c64389b4b27f009c542196476a550a44b8b12e510751e6"} Nov 03 22:12:35.989829 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.989625 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"5ec827c8af95b28c0a3c6babc984ddef6d6da83bd0d6d3c52c6075e7c0fb2000"} Nov 03 22:12:35.989829 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.989655 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"610e6111a9cf0c37e51462730d931940cf2688b38e09fc9c5dd292b02e173169"} Nov 03 22:12:35.989829 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.989683 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"0f672d74ab3d154bf61838dd3fb14639940c85043caf8633b1cd2d2175c6892b"} Nov 03 22:12:35.989829 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.989710 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"dcc29e1f9a4f209355f47336cacbdc2cec23f4bf10e420e3d4231bd7a49fed72"} Nov 03 22:12:35.992654 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:35.992632 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qphlr" event={"ID":"52a290a0-8456-4ca6-afa3-8324f7c76cc0","Type":"ContainerStarted","Data":"0773b4f4f57352d4051d8fefd97290709157f658da6ac964d44d405a5181e458"} Nov 03 22:12:36.069036 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:36.068985 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qphlr" podStartSLOduration=8.510652613 podStartE2EDuration="18.068972778s" totalImagesPullingTime="9.558320165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:36.068927928 +0000 UTC m=+17.861056392" watchObservedRunningTime="2026-11-03 22:12:36.068972778 +0000 UTC m=+17.861101242" Nov 03 22:12:36.424242 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:36.424161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:36.424242 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:36.424201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:36.424424 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.424301 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:36.424424 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.424306 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:36.424424 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.424361 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret podName:17608ecb-fd95-49bf-a939-b2c1587e4fa1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.424348068 +0000 UTC m=+34.216476511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret") pod "global-pull-secret-syncer-nvbcj" (UID: "17608ecb-fd95-49bf-a939-b2c1587e4fa1") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:36.424424 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.424375 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs podName:a04401b8-ddb6-4fed-bac2-2526d011a0ad nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.424369355 +0000 UTC m=+34.216497797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs") pod "network-metrics-daemon-pwbkp" (UID: "a04401b8-ddb6-4fed-bac2-2526d011a0ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:36.524656 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:36.524610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:36.524822 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.524773 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:36.524822 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.524792 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:36.524822 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.524801 2581 projected.go:196] Error preparing data for projected volume kube-api-access-p6gdd for pod openshift-network-diagnostics/network-check-target-xpshj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:36.524921 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.524846 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd podName:a6ad98a8-5e26-471c-a5a0-0ea41c026e1c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.524833563 +0000 UTC m=+34.316962009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6gdd" (UniqueName: "kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd") pod "network-check-target-xpshj" (UID: "a6ad98a8-5e26-471c-a5a0-0ea41c026e1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:36.732326 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:36.732243 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:36.732326 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:36.732280 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:36.732868 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.732371 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:36.732868 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:36.732467 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:37.731838 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:37.731752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:37.731987 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:37.731861 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:37.999700 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:37.999606 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"12e841d5aea8bd67ac9756d66b246c8a820c964061f29e44f13b58654bfdb832"} Nov 03 22:12:38.733137 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:38.733099 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:38.733333 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:38.733216 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:38.733333 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:38.733266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:38.733448 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:38.733392 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:39.731415 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:39.731378 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:39.732085 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:39.731482 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:40.007371 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.007282 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" event={"ID":"e55cd659-eeee-4fbf-b11b-c376d306a652","Type":"ContainerStarted","Data":"a2f2f02a39a15ae1fb4456233f7af3ba3a5e5178af9582d18fbf638686b4c60d"} Nov 03 22:12:40.007539 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.007524 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:40.024175 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.023000 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:40.103216 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.103169 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" podStartSLOduration=7.931326952 podStartE2EDuration="22.103154823s" totalImagesPullingTime="14.171827871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:40.102822903 +0000 UTC m=+21.894951366" watchObservedRunningTime="2026-11-03 22:12:40.103154823 +0000 UTC m=+21.895283307" Nov 03 22:12:40.732780 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.732743 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:40.733512 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:40.733473 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:40.734031 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.732748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:40.734277 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:40.734257 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:40.944631 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.944599 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:40.944836 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.944742 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:40.945164 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:40.945151 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:41.010217 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.010134 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:41.010603 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.010583 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:41.025858 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.025835 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:41.567266 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.567046 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:41.721172 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.721071 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpshj"] Nov 03 22:12:41.721319 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.721173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:41.721319 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:41.721253 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:41.723518 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.723490 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pwbkp"] Nov 03 22:12:41.723654 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.723612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:41.723751 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:41.723712 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:41.726389 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.726363 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nvbcj"] Nov 03 22:12:41.726491 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:41.726466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:41.726580 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:41.726560 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:43.731725 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:43.731690 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:43.732049 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:43.731733 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:43.732049 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:43.731682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:43.732049 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:43.731822 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:43.732049 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:43.731874 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:43.732049 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:43.731975 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:45.732149 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:45.732115 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:45.732579 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:45.732201 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:45.732579 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:45.732212 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:45.732579 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:45.732226 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:45.732579 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:45.732297 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:45.732579 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:45.732387 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:47.731409 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:47.731381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:47.731823 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:47.731466 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nvbcj" podUID="17608ecb-fd95-49bf-a939-b2c1587e4fa1" Nov 03 22:12:47.731823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:47.731542 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:47.731823 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:47.731618 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpshj" podUID="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" Nov 03 22:12:47.731823 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:47.731652 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:47.731823 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:47.731707 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwbkp" podUID="a04401b8-ddb6-4fed-bac2-2526d011a0ad" Nov 03 22:12:48.036782 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.036684 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeReady" Nov 03 22:12:48.036926 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.036888 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:12:48.096556 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.096526 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4hrjm"] Nov 03 22:12:48.101858 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.101822 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9dqrx"] Nov 03 22:12:48.102024 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.101935 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.105706 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.105687 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.121474 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.121448 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9dqrx"] Nov 03 22:12:48.123512 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.123477 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:12:48.123885 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.123856 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:12:48.124044 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.124029 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6vzq\"" Nov 03 22:12:48.124805 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.124782 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4hrjm"] Nov 03 22:12:48.126303 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.126273 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:12:48.134398 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.134378 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:12:48.202096 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.202059 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c88bd7-b717-4197-b99c-4060840ef9d5-tmp-dir\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.202288 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.202115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgjk\" (UniqueName: \"kubernetes.io/projected/32c88bd7-b717-4197-b99c-4060840ef9d5-kube-api-access-fpgjk\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.202288 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.202241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.202288 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.202278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4fl\" (UniqueName: \"kubernetes.io/projected/9156dd68-5a3f-49fd-a004-b736b2dcabce-kube-api-access-9c4fl\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.202457 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.202338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32c88bd7-b717-4197-b99c-4060840ef9d5-config-volume\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.202457 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.202396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9156dd68-5a3f-49fd-a004-b736b2dcabce-cert\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.224416 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.224380 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:12:48.224545 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.224458 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svbdj\"" Nov 03 22:12:48.302790 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.302675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.302790 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.302743 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4fl\" (UniqueName: \"kubernetes.io/projected/9156dd68-5a3f-49fd-a004-b736b2dcabce-kube-api-access-9c4fl\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.302790 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.302769 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32c88bd7-b717-4197-b99c-4060840ef9d5-config-volume\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.303063 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.302824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9156dd68-5a3f-49fd-a004-b736b2dcabce-cert\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.303063 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.302844 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:48.303063 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.302915 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:48.802896208 +0000 UTC m=+30.595024649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:12:48.303063 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.302955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c88bd7-b717-4197-b99c-4060840ef9d5-tmp-dir\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.303063 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.302993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgjk\" (UniqueName: \"kubernetes.io/projected/32c88bd7-b717-4197-b99c-4060840ef9d5-kube-api-access-fpgjk\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.303313 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.303291 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c88bd7-b717-4197-b99c-4060840ef9d5-tmp-dir\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.303446 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.303428 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32c88bd7-b717-4197-b99c-4060840ef9d5-config-volume\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.306811 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.306787 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9156dd68-5a3f-49fd-a004-b736b2dcabce-cert\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.331077 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.331044 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgjk\" (UniqueName: \"kubernetes.io/projected/32c88bd7-b717-4197-b99c-4060840ef9d5-kube-api-access-fpgjk\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.331250 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.331226 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4fl\" (UniqueName: \"kubernetes.io/projected/9156dd68-5a3f-49fd-a004-b736b2dcabce-kube-api-access-9c4fl\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.420393 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.420336 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:48.699105 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.699037 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkl88"] Nov 03 22:12:48.705024 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.705001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.722424 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.722396 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkl88"] Nov 03 22:12:48.734038 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.734011 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:12:48.734438 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.734096 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwzw5\"" Nov 03 22:12:48.734768 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.734752 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:12:48.734881 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.734863 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:12:48.735043 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.735030 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:12:48.744442 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.744419 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9dqrx"] Nov 03 22:12:48.750141 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:48.750109 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9156dd68_5a3f_49fd_a004_b736b2dcabce.slice/crio-6cae49f2597fba57d633d2aa1e2c0afb699a58d9cf612dcd55ae1afad6c394aa WatchSource:0}: Error finding container 6cae49f2597fba57d633d2aa1e2c0afb699a58d9cf612dcd55ae1afad6c394aa: Status 404 returned error can't find the container with id 6cae49f2597fba57d633d2aa1e2c0afb699a58d9cf612dcd55ae1afad6c394aa Nov 03 22:12:48.785119 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.785092 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q8k65-h5z45"] Nov 03 22:12:48.807642 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.807611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:48.807771 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.807653 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.807771 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.807673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2b4d5ca-dead-4cca-b082-91264d877d59-crio-socket\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.807855 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.807827 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:48.807899 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.807889 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:49.807876251 +0000 UTC m=+31.600004693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:12:48.807936 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.807924 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b4d5ca-dead-4cca-b082-91264d877d59-data-volume\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.807968 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.807942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pxb\" (UniqueName: \"kubernetes.io/projected/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-api-access-l2pxb\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.808003 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.807984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.812626 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.812586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:48.815489 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.815448 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q8k65-h5z45"] Nov 03 22:12:48.908758 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2b4d5ca-dead-4cca-b082-91264d877d59-crio-socket\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.908908 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b4d5ca-dead-4cca-b082-91264d877d59-data-volume\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.908908 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pxb\" (UniqueName: \"kubernetes.io/projected/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-api-access-l2pxb\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.908908 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.908908 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwh8\" (UniqueName: \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:48.908908 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:48.909112 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908921 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2b4d5ca-dead-4cca-b082-91264d877d59-crio-socket\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.909112 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.908970 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:48.909112 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.908999 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:48.909112 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.909061 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:49.409046501 +0000 UTC m=+31.201174943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:48.909243 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.909121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:48.909243 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.909150 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.909243 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.909214 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:48.909243 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.909223 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b4d5ca-dead-4cca-b082-91264d877d59-data-volume\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:48.909357 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:48.909269 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:49.409256597 +0000 UTC m=+31.201385042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : configmap "kube-rbac-proxy" not found Nov 03 22:12:48.925266 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.925218 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:12:48.925366 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.925268 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d79qx\"" Nov 03 22:12:48.940004 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.939977 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:12:48.942134 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:48.942105 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pxb\" (UniqueName: \"kubernetes.io/projected/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-api-access-l2pxb\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:49.010325 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.010242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.010325 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.010276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.010513 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.010339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwh8\" (UniqueName: \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.010513 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.010357 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.010746 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.010708 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.011036 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.011015 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.011227 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.011212 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.026031 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.025997 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9dqrx" event={"ID":"9156dd68-5a3f-49fd-a004-b736b2dcabce","Type":"ContainerStarted","Data":"6cae49f2597fba57d633d2aa1e2c0afb699a58d9cf612dcd55ae1afad6c394aa"} Nov 03 22:12:49.035338 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.035317 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwh8\" (UniqueName: \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.136536 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.136495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:49.414109 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.414033 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:49.414281 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:49.414112 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:49.414281 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.414132 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:49.414281 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:49.414182 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.41416695 +0000 UTC m=+32.206295392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : configmap "kube-rbac-proxy" not found Nov 03 22:12:49.414455 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:49.414288 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:49.414455 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:49.414342 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.414327332 +0000 UTC m=+32.206455777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:49.451551 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.451519 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q8k65-h5z45"] Nov 03 22:12:49.455422 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:49.455392 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacca86e6_473d_442d_9ca9_8ba949d8caa9.slice/crio-34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3 WatchSource:0}: Error finding container 34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3: Status 404 returned error can't find the container with id 34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3 Nov 03 22:12:49.729091 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.728990 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fztdk"] Nov 03 22:12:49.763149 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.763115 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fztdk"] Nov 03 22:12:49.763601 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.763244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:49.763601 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.763252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.763601 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.763462 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:49.763755 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.763739 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:49.779848 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.779827 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:12:49.780178 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780035 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:12:49.780178 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780087 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 22:12:49.780178 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780130 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 22:12:49.780178 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780138 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 22:12:49.780178 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780135 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 22:12:49.780178 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780167 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mq5fv\"" Nov 03 22:12:49.780515 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780281 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9h6fz\"" Nov 03 22:12:49.780515 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780351 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zdncz\"" Nov 03 22:12:49.780515 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.780428 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:12:49.793653 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.793613 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9wkj9\"" Nov 03 22:12:49.793653 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.793631 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:12:49.818186 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.818153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbqkh\" (UniqueName: \"kubernetes.io/projected/73539f96-a0bb-45c0-9ec0-596d7721996c-kube-api-access-dbqkh\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.818345 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.818220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:49.818345 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.818320 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-cabundle\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.818441 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:49.818368 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:49.818441 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.818381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-key\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.818441 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:49.818436 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:51.818419469 +0000 UTC m=+33.610547914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:12:49.919949 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.919917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbqkh\" (UniqueName: \"kubernetes.io/projected/73539f96-a0bb-45c0-9ec0-596d7721996c-kube-api-access-dbqkh\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.920138 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.920027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-cabundle\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.920138 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.920074 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-key\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.921019 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.920992 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-cabundle\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.923500 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.923468 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-key\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:49.945878 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:49.945849 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbqkh\" (UniqueName: \"kubernetes.io/projected/73539f96-a0bb-45c0-9ec0-596d7721996c-kube-api-access-dbqkh\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:50.030174 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.030093 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8k65-h5z45" event={"ID":"acca86e6-473d-442d-9ca9-8ba949d8caa9","Type":"ContainerStarted","Data":"34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3"} Nov 03 22:12:50.156617 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.156576 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:50.168911 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.168873 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw"] Nov 03 22:12:50.185232 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.185205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" Nov 03 22:12:50.198926 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.198243 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw"] Nov 03 22:12:50.288970 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.288885 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:50.288970 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.288959 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 22:12:50.291178 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.291154 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjknz\"" Nov 03 22:12:50.324628 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.324592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm74n\" (UniqueName: \"kubernetes.io/projected/6602f77f-4f03-409b-8174-3ae6d73f6420-kube-api-access-gm74n\") pod \"migrator-66df5d496b-6mxpw\" (UID: \"6602f77f-4f03-409b-8174-3ae6d73f6420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" Nov 03 22:12:50.426028 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.425984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:50.426227 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.426053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm74n\" (UniqueName: \"kubernetes.io/projected/6602f77f-4f03-409b-8174-3ae6d73f6420-kube-api-access-gm74n\") pod \"migrator-66df5d496b-6mxpw\" (UID: \"6602f77f-4f03-409b-8174-3ae6d73f6420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" Nov 03 22:12:50.426227 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.426124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:50.426342 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:50.426295 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:50.426398 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:50.426355 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.426338504 +0000 UTC m=+34.218466951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:50.426780 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:50.426750 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:50.426880 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:50.426810 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.426793769 +0000 UTC m=+34.218922213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : configmap "kube-rbac-proxy" not found Nov 03 22:12:50.452006 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.451971 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm74n\" (UniqueName: \"kubernetes.io/projected/6602f77f-4f03-409b-8174-3ae6d73f6420-kube-api-access-gm74n\") pod \"migrator-66df5d496b-6mxpw\" (UID: \"6602f77f-4f03-409b-8174-3ae6d73f6420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" Nov 03 22:12:50.493027 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.492988 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fztdk"] Nov 03 22:12:50.502587 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:50.498579 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73539f96_a0bb_45c0_9ec0_596d7721996c.slice/crio-07da56523951ca288bd922e19de9a2a9211ae2ebf88ecd5a0c5abd9bc841dca1 WatchSource:0}: Error finding container 07da56523951ca288bd922e19de9a2a9211ae2ebf88ecd5a0c5abd9bc841dca1: Status 404 returned error can't find the container with id 07da56523951ca288bd922e19de9a2a9211ae2ebf88ecd5a0c5abd9bc841dca1 Nov 03 22:12:50.524764 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.524729 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" Nov 03 22:12:50.873498 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:50.873419 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw"] Nov 03 22:12:50.877585 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:50.877543 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6602f77f_4f03_409b_8174_3ae6d73f6420.slice/crio-ae84f6f52b777d1c7fa2233f92e4e6e60729a8b405138decb04d5129b510b0ac WatchSource:0}: Error finding container ae84f6f52b777d1c7fa2233f92e4e6e60729a8b405138decb04d5129b510b0ac: Status 404 returned error can't find the container with id ae84f6f52b777d1c7fa2233f92e4e6e60729a8b405138decb04d5129b510b0ac Nov 03 22:12:51.035820 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:51.035486 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" event={"ID":"73539f96-a0bb-45c0-9ec0-596d7721996c","Type":"ContainerStarted","Data":"07da56523951ca288bd922e19de9a2a9211ae2ebf88ecd5a0c5abd9bc841dca1"} Nov 03 22:12:51.037284 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:51.037142 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" event={"ID":"6602f77f-4f03-409b-8174-3ae6d73f6420","Type":"ContainerStarted","Data":"ae84f6f52b777d1c7fa2233f92e4e6e60729a8b405138decb04d5129b510b0ac"} Nov 03 22:12:51.838831 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:51.838799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:51.839119 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:51.838981 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:51.839119 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:51.839058 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.839037857 +0000 UTC m=+37.631166313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:12:52.444492 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.444456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.444506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:52.444583 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:52.444607 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:52.444648 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.44463035 +0000 UTC m=+38.236758804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : configmap "kube-rbac-proxy" not found Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:52.444666 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs podName:a04401b8-ddb6-4fed-bac2-2526d011a0ad nodeName:}" failed. No retries permitted until 2026-11-03 22:13:24.444658005 +0000 UTC m=+66.236786451 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs") pod "network-metrics-daemon-pwbkp" (UID: "a04401b8-ddb6-4fed-bac2-2526d011a0ad") : secret "metrics-daemon-secret" not found Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.444682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.444745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:52.444846 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:52.444956 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:52.444891 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls podName:b2b4d5ca-dead-4cca-b082-91264d877d59 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.444880914 +0000 UTC m=+38.237009370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkl88" (UID: "b2b4d5ca-dead-4cca-b082-91264d877d59") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:52.447236 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.447208 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:52.543136 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.543104 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:52.545436 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.545409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:52.548253 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.548224 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:52.815357 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.815330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:52.992103 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:52.992030 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nvbcj"] Nov 03 22:12:52.995733 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:52.995693 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17608ecb_fd95_49bf_a939_b2c1587e4fa1.slice/crio-9d2b8d4b49f6fb1836cbb4f32773ffa180e1ca3963ff3cf0dd239ec45a4abbda WatchSource:0}: Error finding container 9d2b8d4b49f6fb1836cbb4f32773ffa180e1ca3963ff3cf0dd239ec45a4abbda: Status 404 returned error can't find the container with id 9d2b8d4b49f6fb1836cbb4f32773ffa180e1ca3963ff3cf0dd239ec45a4abbda Nov 03 22:12:53.043779 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.043744 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8k65-h5z45" event={"ID":"acca86e6-473d-442d-9ca9-8ba949d8caa9","Type":"ContainerStarted","Data":"4f95cd10477843871f654653bfdaa2b63967e4cb93a32e115a476078d9c9310a"} Nov 03 22:12:53.044836 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.044812 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nvbcj" event={"ID":"17608ecb-fd95-49bf-a939-b2c1587e4fa1","Type":"ContainerStarted","Data":"9d2b8d4b49f6fb1836cbb4f32773ffa180e1ca3963ff3cf0dd239ec45a4abbda"} Nov 03 22:12:53.046241 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.046221 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9dqrx" event={"ID":"9156dd68-5a3f-49fd-a004-b736b2dcabce","Type":"ContainerStarted","Data":"398cf7723dbd8d1cda3f7abe478ca9349e2dc9519b26dadd70e9bc385b8d5599"} Nov 03 22:12:53.047596 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.047576 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" event={"ID":"6602f77f-4f03-409b-8174-3ae6d73f6420","Type":"ContainerStarted","Data":"defbbad24c916c5c7a3865eb72b601148886a9e4efe1389b8609a84d189ebb4d"} Nov 03 22:12:53.086278 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.086232 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-q8k65-h5z45" podStartSLOduration=2.563645413 podStartE2EDuration="5.086219318s" totalImagesPullingTime="2.522573905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:53.085912695 +0000 UTC m=+34.878041159" watchObservedRunningTime="2026-11-03 22:12:53.086219318 +0000 UTC m=+34.878347782" Nov 03 22:12:53.126879 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.126837 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" podStartSLOduration=1.332486761 podStartE2EDuration="3.126824601s" totalImagesPullingTime="1.79433784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:53.126635054 +0000 UTC m=+34.918763518" watchObservedRunningTime="2026-11-03 22:12:53.126824601 +0000 UTC m=+34.918953065" Nov 03 22:12:53.165065 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.165038 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpshj"] Nov 03 22:12:53.169832 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:53.169793 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9dqrx" podStartSLOduration=1.951932977 podStartE2EDuration="5.169780291s" totalImagesPullingTime="3.217847314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:53.169513583 +0000 UTC m=+34.961642046" watchObservedRunningTime="2026-11-03 22:12:53.169780291 +0000 UTC m=+34.961908754" Nov 03 22:12:53.275502 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:53.275434 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ad98a8_5e26_471c_a5a0_0ea41c026e1c.slice/crio-0bd09b8b09d2f4eae5953cd491109d83cf16569fd08c2bf1cf6e428c72c14a47 WatchSource:0}: Error finding container 0bd09b8b09d2f4eae5953cd491109d83cf16569fd08c2bf1cf6e428c72c14a47: Status 404 returned error can't find the container with id 0bd09b8b09d2f4eae5953cd491109d83cf16569fd08c2bf1cf6e428c72c14a47 Nov 03 22:12:54.052563 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:54.052521 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" event={"ID":"73539f96-a0bb-45c0-9ec0-596d7721996c","Type":"ContainerStarted","Data":"da2cbf7eb2795bc5117b879309ca1b793ffe4afff1a7136334a0095c80e31a4e"} Nov 03 22:12:54.055069 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:54.055029 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpshj" event={"ID":"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c","Type":"ContainerStarted","Data":"0bd09b8b09d2f4eae5953cd491109d83cf16569fd08c2bf1cf6e428c72c14a47"} Nov 03 22:12:54.097826 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:54.097190 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" podStartSLOduration=2.283910776 podStartE2EDuration="5.097176551s" totalImagesPullingTime="2.813265775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:54.096012241 +0000 UTC m=+35.888140735" watchObservedRunningTime="2026-11-03 22:12:54.097176551 +0000 UTC m=+35.889305055" Nov 03 22:12:55.883438 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:55.883394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:55.883927 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:55.883555 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:55.883927 ip-10-0-128-81 kubenswrapper[2581]: E1103 22:12:55.883621 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:03.883604837 +0000 UTC m=+45.675733280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:12:55.888880 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:55.888776 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97blh_63ada7c0-b6ab-4ba2-bc6d-990b007311a1/dns-node-resolver/0.log" Nov 03 22:12:56.488549 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:56.488507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:56.488768 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:56.488610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:56.489323 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:56.489294 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:56.493390 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:56.493352 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:56.520566 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:56.520537 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:56.887359 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:56.887307 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkl88"] Nov 03 22:12:56.890547 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:56.890525 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g7nfl_e5eb73d7-8dd5-4e38-b312-9fd9f4881762/node-ca/0.log" Nov 03 22:12:56.913015 ip-10-0-128-81 kubenswrapper[2581]: W1103 22:12:56.912986 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b4d5ca_dead_4cca_b082_91264d877d59.slice/crio-b1d27604a03d0165b240c85c3938698e4958789e1974fabb47b434fcb5ac3155 WatchSource:0}: Error finding container b1d27604a03d0165b240c85c3938698e4958789e1974fabb47b434fcb5ac3155: Status 404 returned error can't find the container with id b1d27604a03d0165b240c85c3938698e4958789e1974fabb47b434fcb5ac3155 Nov 03 22:12:57.065401 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:57.065374 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpshj" event={"ID":"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c","Type":"ContainerStarted","Data":"ae4d66124c818d585c9bdcf8f4a7b2c5a1f2ce775ddf398cfd57da78d076c10b"} Nov 03 22:12:57.065853 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:57.065832 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:57.067511 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:57.067492 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkl88" event={"ID":"b2b4d5ca-dead-4cca-b082-91264d877d59","Type":"ContainerStarted","Data":"14ceb1e2c2790848bc60e1e63e64396d79d841bfc15540dade10490cd46d64c1"} Nov 03 22:12:57.067605 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:57.067531 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkl88" event={"ID":"b2b4d5ca-dead-4cca-b082-91264d877d59","Type":"ContainerStarted","Data":"b1d27604a03d0165b240c85c3938698e4958789e1974fabb47b434fcb5ac3155"} Nov 03 22:12:57.111503 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:57.111447 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xpshj" podStartSLOduration=35.465163019 podStartE2EDuration="39.111432118s" totalImagesPullingTime="3.646269099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:57.109819879 +0000 UTC m=+38.901948344" watchObservedRunningTime="2026-11-03 22:12:57.111432118 +0000 UTC m=+38.903560583" Nov 03 22:12:57.491773 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:57.491523 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9dqrx_9156dd68-5a3f-49fd-a004-b736b2dcabce/serve-healthcheck-canary/0.log" Nov 03 22:12:58.088978 ip-10-0-128-81 kubenswrapper[2581]: I1103 22:12:58.088943 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6mxpw_6602f77f-4f03-409b-8174-3ae6d73f6420/migrator/0.log" Nov 03 22:12:58.506564 ip-10-0-128-81 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:12:58.517169 ip-10-0-128-81 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:12:58.517355 ip-10-0-128-81 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:12:58.517407 ip-10-0-128-81 systemd[1]: kubelet.service: Consumed 2.430s CPU time. Nov 03 22:12:58.525486 ip-10-0-128-81 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:12:58.617004 ip-10-0-128-81 kubenswrapper[4625]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:58.617004 ip-10-0-128-81 kubenswrapper[4625]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:12:58.617004 ip-10-0-128-81 kubenswrapper[4625]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:58.617004 ip-10-0-128-81 kubenswrapper[4625]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:58.620460 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620331 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.620460 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620459 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.620460 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620465 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620470 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620475 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620480 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620484 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620488 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620492 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620496 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620500 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620504 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620508 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620513 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620517 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620521 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620525 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620529 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620534 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620538 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620544 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.620602 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620548 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620552 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620556 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620560 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620565 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620569 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620573 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620577 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620581 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620586 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620590 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620594 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620598 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620603 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620607 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620611 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620616 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620620 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620624 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620628 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.621177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620633 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620637 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620641 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620645 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620649 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620654 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620658 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620662 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620668 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620675 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620680 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620685 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620689 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620693 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620698 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620702 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620706 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620711 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620728 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.621771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620733 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620738 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620742 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620746 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620750 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620756 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620760 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620764 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620769 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620773 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620777 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620781 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620785 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620789 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620793 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620797 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620801 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620835 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620844 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620850 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.622238 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620855 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620860 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620864 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620868 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620876 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620880 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620884 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620889 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620893 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620898 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620902 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620906 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620910 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620914 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620919 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620925 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620931 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620936 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620942 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.622731 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620947 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620951 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620955 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620959 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620963 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620967 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.620973 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622141 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622149 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622154 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622158 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622162 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622166 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622171 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622175 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622179 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622183 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622188 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622192 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622196 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.623189 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622201 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622205 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622209 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622213 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622217 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622221 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622225 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622229 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622233 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622237 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622241 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622245 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622259 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622264 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622268 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622271 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622275 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622279 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622283 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.623668 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622287 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622291 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622295 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622299 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622303 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622307 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622311 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622315 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622319 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622323 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622327 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622331 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622335 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622340 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622344 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622348 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622352 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622356 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622360 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622364 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.624149 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622367 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622371 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622375 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622379 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622383 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622387 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622398 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622402 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622406 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622410 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622414 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622418 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622422 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622426 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622430 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622434 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622439 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622443 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622447 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622451 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.624672 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622455 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622459 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622466 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622471 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622476 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622480 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622484 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622488 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622492 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622496 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622500 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622504 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622508 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622512 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622516 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622520 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622524 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622528 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622531 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.625200 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622542 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622546 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622550 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622554 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622558 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622562 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622566 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622570 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622574 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622580 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622588 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622593 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622597 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622602 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.622606 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623426 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623433 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623438 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623442 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.625680 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623447 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623451 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623455 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623459 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623463 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623467 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623471 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623475 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623479 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623483 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623487 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623490 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623494 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623499 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623503 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623508 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623512 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623516 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623520 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623524 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.626185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623532 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623536 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623540 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623544 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623548 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623552 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623556 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623561 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623566 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623570 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623574 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623578 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623582 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623585 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623589 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623593 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623597 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623601 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623606 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.626663 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623612 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623616 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623621 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623626 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623630 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623634 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623638 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623644 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623650 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623654 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623658 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623662 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623667 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623673 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623677 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623682 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623703 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623710 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623729 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.627262 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623733 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623737 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623742 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623746 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623750 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623754 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623758 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623763 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623767 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623770 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623774 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623778 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623783 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623787 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623791 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623795 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623799 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623802 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623806 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623810 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.627751 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623814 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623821 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623825 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623829 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623833 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623837 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623843 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623847 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623851 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623855 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623859 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623863 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623867 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623871 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623877 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623883 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623887 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623891 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623895 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.628236 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623899 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623905 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623909 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623913 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.623917 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624037 4625 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624054 4625 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624062 4625 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624069 4625 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624076 4625 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624081 4625 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624087 4625 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624094 4625 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624099 4625 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624105 4625 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624112 4625 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624117 4625 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624122 4625 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624126 4625 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624132 4625 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624137 4625 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624142 4625 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624147 4625 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:12:58.628853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624153 4625 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624158 4625 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624163 4625 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624168 4625 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624173 4625 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624179 4625 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624183 4625 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624188 4625 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624193 4625 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624197 4625 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624202 4625 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624207 4625 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624212 4625 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624216 4625 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624223 4625 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624228 4625 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624232 4625 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624236 4625 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624241 4625 flags.go:64] FLAG: --enable-server="true" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624245 4625 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624252 4625 flags.go:64] FLAG: --event-burst="100" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624256 4625 flags.go:64] FLAG: --event-qps="50" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624260 4625 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624266 4625 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:12:58.629811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624270 4625 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624278 4625 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624282 4625 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624287 4625 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624293 4625 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624298 4625 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624302 4625 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624306 4625 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624317 4625 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624322 4625 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624327 4625 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624331 4625 flags.go:64] FLAG: --feature-gates="" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624338 4625 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624342 4625 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624347 4625 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624370 4625 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624375 4625 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624380 4625 flags.go:64] FLAG: --help="false" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624384 4625 flags.go:64] FLAG: --hostname-override="ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624389 4625 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624394 4625 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624399 4625 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624404 4625 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:12:58.630535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624410 4625 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624415 4625 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624419 4625 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624424 4625 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624429 4625 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624433 4625 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624438 4625 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624443 4625 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624448 4625 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624452 4625 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624457 4625 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624462 4625 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624467 4625 flags.go:64] FLAG: --lock-file="" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624473 4625 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624478 4625 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624482 4625 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624491 4625 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624495 4625 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624500 4625 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624505 4625 flags.go:64] FLAG: --logging-format="text" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624510 4625 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624515 4625 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624520 4625 flags.go:64] FLAG: --manifest-url="" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624524 4625 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624531 4625 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:12:58.631103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624536 4625 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624542 4625 flags.go:64] FLAG: --max-pods="110" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624546 4625 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624551 4625 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624556 4625 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624561 4625 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624565 4625 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624570 4625 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624575 4625 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624585 4625 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624590 4625 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624595 4625 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624600 4625 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624604 4625 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624608 4625 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624613 4625 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624618 4625 flags.go:64] FLAG: --port="10250" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624623 4625 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624628 4625 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de017e703e244506" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624633 4625 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624640 4625 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624645 4625 flags.go:64] FLAG: --register-node="true" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624650 4625 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624656 4625 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624660 4625 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624665 4625 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:12:58.631708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624670 4625 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624676 4625 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624681 4625 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624686 4625 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624691 4625 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624695 4625 flags.go:64] FLAG: --runonce="false" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624700 4625 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624705 4625 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624709 4625 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624727 4625 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624733 4625 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624737 4625 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624742 4625 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624747 4625 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624752 4625 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624757 4625 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624761 4625 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624766 4625 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624770 4625 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624775 4625 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624780 4625 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624788 4625 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624793 4625 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624797 4625 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624804 4625 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:12:58.632347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624808 4625 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624814 4625 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624819 4625 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624824 4625 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624828 4625 flags.go:64] FLAG: --v="2" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624835 4625 flags.go:64] FLAG: --version="false" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624841 4625 flags.go:64] FLAG: --vmodule="" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624862 4625 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.624867 4625 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625079 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625084 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625089 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625093 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625097 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625101 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625106 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625110 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625116 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625122 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625127 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625131 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.632989 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625135 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625139 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625144 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625148 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625152 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625156 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625160 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625167 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625171 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625175 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625179 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625183 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625187 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625192 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625196 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625200 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625205 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625209 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625213 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625217 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.633501 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625222 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625226 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625230 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625235 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625239 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625243 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625247 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625251 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625255 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625259 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625263 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625267 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625272 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625276 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625280 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625284 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625288 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625292 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625296 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625303 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.634040 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625307 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625311 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625316 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625320 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625324 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625328 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625332 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625337 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625341 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625345 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625349 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625353 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625357 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625362 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625366 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625370 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625375 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625382 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625388 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.634555 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625393 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625398 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625414 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625419 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625423 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625428 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625432 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625436 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625440 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625445 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625449 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625453 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625460 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625464 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625469 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625473 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625478 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625482 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625486 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.635038 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625490 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625495 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625499 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625503 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625507 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625511 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625515 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625519 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625522 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625526 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625530 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625535 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625539 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625543 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625547 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625551 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.635592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.625589 4625 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625661 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625669 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625675 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625680 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625687 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625691 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625695 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625699 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625706 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625710 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625727 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625732 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625736 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625740 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625744 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625748 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625752 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.636061 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625756 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625761 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625765 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625769 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625773 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625777 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625782 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625786 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625790 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625794 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625797 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625802 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625806 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625810 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625813 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625817 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625822 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625826 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625830 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.636658 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625835 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625839 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625843 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625847 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625853 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625857 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625861 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625865 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625869 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625873 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625879 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625885 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625889 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625893 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625898 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625902 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625906 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625910 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625914 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625918 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.637205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625922 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625929 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625933 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625937 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625941 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625945 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625949 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625953 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625957 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625961 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625965 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625969 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625975 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625979 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625983 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625987 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625992 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.625997 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626001 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626005 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.637869 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626009 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626013 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626017 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626021 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626025 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626029 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626033 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626038 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626042 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626046 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626050 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626054 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626058 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626062 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626066 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626070 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626074 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626078 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626082 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626086 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.638777 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626090 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626094 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626098 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626102 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626108 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626112 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626116 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626120 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626127 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.626132 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.631149 4625 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.631164 4625 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.631182 4625 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.631187 4625 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631246 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631252 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631255 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631259 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631262 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631264 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.639496 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631267 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631270 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631273 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631275 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631278 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631281 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631284 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631286 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631289 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631292 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631295 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631297 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631300 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631302 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631305 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631308 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631310 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631313 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631316 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631318 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.640162 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631321 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631324 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631327 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631330 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631332 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631335 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631338 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631341 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631344 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631346 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631349 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631351 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631354 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631356 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631359 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631362 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631366 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631370 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631372 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.640811 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631376 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631379 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631382 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631384 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631388 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631392 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631396 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631398 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631401 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631403 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631406 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631409 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631411 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631414 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631417 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631420 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631422 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631425 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631427 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.641290 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631430 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631432 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631435 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631437 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631440 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631442 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631445 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631447 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631450 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631452 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631455 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631457 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631460 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631462 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631465 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631468 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631470 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631473 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631476 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.641804 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631478 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631481 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631483 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631486 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631488 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631491 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631493 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631496 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631499 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631501 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631504 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631506 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631509 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631512 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631514 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631517 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631519 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631522 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631525 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631527 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.642253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631530 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631532 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631534 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.631563 4625 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631702 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631707 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631710 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631728 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631731 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631736 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631741 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631745 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631748 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631751 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631754 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631757 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.642954 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631760 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631762 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631765 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631767 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631770 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631772 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631775 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631778 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631780 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631783 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631786 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631788 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631790 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631793 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631796 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631798 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631801 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631804 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631806 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.643528 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631809 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631811 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631814 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631816 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631819 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631821 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631824 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631827 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631831 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631835 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631838 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631841 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631844 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631846 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631849 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631852 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631854 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631857 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631859 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.644208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631862 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631864 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631867 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631869 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631871 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631874 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631877 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631879 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631881 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631884 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631887 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631890 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631893 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631895 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631898 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631900 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631903 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631906 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631908 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631911 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.644770 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631915 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631918 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631921 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631924 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631927 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631930 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631932 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631935 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631938 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631941 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631943 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631946 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631948 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631951 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631953 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631956 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631958 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631961 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631963 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631966 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.645301 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631969 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631971 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631974 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631976 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631979 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631982 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631984 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631986 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631989 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631992 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631994 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.631997 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632001 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632004 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632007 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632010 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.645827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.632033 4625 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632067 4625 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632072 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632075 4625 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632078 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632081 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632084 4625 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632087 4625 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632091 4625 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632094 4625 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632097 4625 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632100 4625 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632103 4625 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632105 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632108 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632110 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632113 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632115 4625 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632118 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:58.646305 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632120 4625 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632123 4625 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632126 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632128 4625 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632131 4625 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632133 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632136 4625 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632138 4625 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632141 4625 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632145 4625 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632147 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632150 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632153 4625 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632156 4625 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632158 4625 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632161 4625 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632163 4625 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632166 4625 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632168 4625 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:58.647248 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632171 4625 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632173 4625 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632176 4625 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632180 4625 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632183 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632185 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632188 4625 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632191 4625 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632193 4625 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632196 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632198 4625 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632201 4625 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632203 4625 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632206 4625 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632208 4625 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632211 4625 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632213 4625 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632216 4625 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632218 4625 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632220 4625 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:58.647771 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632223 4625 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632225 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632229 4625 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632232 4625 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632234 4625 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632237 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632240 4625 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632243 4625 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632246 4625 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632248 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632252 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632254 4625 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632257 4625 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632259 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632262 4625 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632264 4625 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632267 4625 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632269 4625 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632272 4625 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632275 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:58.648242 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632277 4625 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632280 4625 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632282 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632285 4625 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632287 4625 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632290 4625 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632292 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632295 4625 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632297 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632300 4625 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632302 4625 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632305 4625 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632307 4625 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632310 4625 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632320 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632323 4625 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632326 4625 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632328 4625 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632331 4625 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632337 4625 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:58.648780 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632340 4625 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632343 4625 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632346 4625 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632367 4625 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632370 4625 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632373 4625 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632376 4625 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632378 4625 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:12:58.632381 4625 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.632439 4625 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.633417 4625 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.633491 4625 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.633685 4625 server.go:1014] "Starting client certificate rotation" Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.633831 4625 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.633878 4625 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:07:18 +0000 UTC" deadline="2028-04-30 05:45:32.345290551 +0000 UTC" Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.633913 4625 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13039h32m33.711380163s" Nov 03 22:12:58.649265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.634241 4625 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:58.649665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.635458 4625 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:58.649665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.637056 4625 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:12:58.649665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.639137 4625 log.go:25] "Validated CRI v1 image API" Nov 03 22:12:58.649665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.639915 4625 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:12:58.649665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.641976 4625 fs.go:135] Filesystem UUIDs: map[0e9f578c-82a3-451b-8bed-3986666ebf75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e3b71187-b4e3-4782-8520-87e0c365672c:/dev/nvme0n1p3] Nov 03 22:12:58.649830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.641998 4625 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00c1e219b17813ca93a7a52190c2fd54cc03e406f50c62db1f35c64096b29a44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00c1e219b17813ca93a7a52190c2fd54cc03e406f50c62db1f35c64096b29a44/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07da56523951ca288bd922e19de9a2a9211ae2ebf88ecd5a0c5abd9bc841dca1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07da56523951ca288bd922e19de9a2a9211ae2ebf88ecd5a0c5abd9bc841dca1/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bd09b8b09d2f4eae5953cd491109d83cf16569fd08c2bf1cf6e428c72c14a47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bd09b8b09d2f4eae5953cd491109d83cf16569fd08c2bf1cf6e428c72c14a47/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eebdb82a457f55b8ffbf3a093e6ebf1efdb444bfcccb2df6f04a8b88c0461a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eebdb82a457f55b8ffbf3a093e6ebf1efdb444bfcccb2df6f04a8b88c0461a7/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f3e4e964af0a63479dfd41c9f4c4e350dc6e1a862b216a6b1a3555eaa860949/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f3e4e964af0a63479dfd41c9f4c4e350dc6e1a862b216a6b1a3555eaa860949/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/548691e3d13dbd44053a2a8716d9dbe685fa1eafa2b3435c26241af81eba8b5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/548691e3d13dbd44053a2a8716d9dbe685fa1eafa2b3435c26241af81eba8b5a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fb1828bfc38c813678aea328f233a195d7ecf8c1ea18ee298404d8a595d604d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fb1828bfc38c813678aea328f233a195d7ecf8c1ea18ee298404d8a595d604d/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cae49f2597fba57d633d2aa1e2c0afb699a58d9cf612dcd55ae1afad6c394aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cae49f2597fba57d633d2aa1e2c0afb699a58d9cf612dcd55ae1afad6c394aa/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e495d17d943b9577b3777b0f9b4282d657d02b3848e28f99fa5b6673e238741/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e495d17d943b9577b3777b0f9b4282d657d02b3848e28f99fa5b6673e238741/userdata/shm major:0 minor:106 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d2b8d4b49f6fb1836cbb4f32773ffa180e1ca3963ff3cf0dd239ec45a4abbda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d2b8d4b49f6fb1836cbb4f32773ffa180e1ca3963ff3cf0dd239ec45a4abbda/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae84f6f52b777d1c7fa2233f92e4e6e60729a8b405138decb04d5129b510b0ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae84f6f52b777d1c7fa2233f92e4e6e60729a8b405138decb04d5129b510b0ac/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1d27604a03d0165b240c85c3938698e4958789e1974fabb47b434fcb5ac3155/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1d27604a03d0165b240c85c3938698e4958789e1974fabb47b434fcb5ac3155/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bff27fde53144f474e8e4ca5cfe745e5eb2a85a6444c7123a61c1c595d4d6c50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bff27fde53144f474e8e4ca5cfe745e5eb2a85a6444c7123a61c1c595d4d6c50/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2630804f37348b7e6d930f4b96f4fdd9d685d00c5f06484f901af01e0cc5015/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2630804f37348b7e6d930f4b96f4fdd9d685d00c5f06484f901af01e0cc5015/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f6162d-1577-465d-83c3-b020607dd4e9/volumes/kubernetes.io~projected/kube-api-access-t2wnh:{mountpoint:/var/lib/kubelet/pods/05f6162d-1577-465d-83c3-b020607dd4e9/volumes/kubernetes.io~projected/kube-api-access-t2wnh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17608ecb-fd95-49bf-a939-b2c1587e4fa1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/17608ecb-fd95-49bf-a939-b2c1587e4fa1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32c88bd7-b717-4197-b99c-4060840ef9d5/volumes/kubernetes.io~projected/kube-api-access-fpgjk:{mountpoint:/var/lib/kubelet/pods/32c88bd7-b717-4197-b99c-4060840ef9d5/volumes/kubernetes.io~projected/kube-api-access-fpgjk major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a290a0-8456-4ca6-afa3-8324f7c76cc0/volumes/kubernetes.io~projected/kube-api-access-jqkqf:{mountpoint:/var/lib/kubelet/pods/52a290a0-8456-4ca6-afa3-8324f7c76cc0/volumes/kubernetes.io~projected/kube-api-access-jqkqf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~projected/kube-api-access-5lnxp:{mountpoint:/var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~projected/kube-api-access-5lnxp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63ada7c0-b6ab-4ba2-bc6d-990b007311a1/volumes/kubernetes.io~projected/kube-api-access-ntqgq:{mountpoint:/var/lib/kubelet/pods/63ada7c0-b6ab-4ba2-bc6d-990b007311a1/volumes/kubernetes.io~projected/kube-api-access-ntqgq major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6602f77f-4f03-409b-8174-3ae6d73f6420/volumes/kubernetes.io~projected/kube-api-access-gm74n:{mountpoint:/var/lib/kubelet/pods/6602f77f-4f03-409b-8174-3ae6d73f6420/volumes/kubernetes.io~projected/kube-api-access-gm74n major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73539f96-a0bb-45c0-9ec0-596d7721996c/volumes/kubernetes.io~projected/kube-api-access-dbqkh:{mountpoint:/var/lib/kubelet/pods/73539f96-a0bb-45c0-9ec0-596d7721996c/volumes/kubernetes.io~projected/kube-api-access-dbqkh major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73539f96-a0bb-45c0-9ec0-596d7721996c/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/73539f96-a0bb-45c0-9ec0-596d7721996c/volumes/kubernetes.io~secret/signing-key major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9156dd68-5a3f-49fd-a004-b736b2dcabce/volumes/kubernetes.io~projected/kube-api-access-9c4fl:{mountpoint:/var/lib/kubelet/pods/9156dd68-5a3f-49fd-a004-b736b2dcabce/volumes/kubernetes.io~projected/kube-api-access-9c4fl major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9156dd68-5a3f-49fd-a004-b736b2dcabce/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9156dd68-5a3f-49fd-a004-b736b2dcabce/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/941fe33c-1af4-4a8c-bbda-6c3756edae7f/volumes/kubernetes.io~projected/kube-api-access-rjvs5:{mountpoint:/var/lib/kubelet/pods/941fe33c-1af4-4a8c-bbda-6c3756edae7f/volumes/kubernetes.io~projected/kube-api-access-rjvs5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a04401b8-ddb6-4fed-bac2-2526d011a0ad/volumes/kubernetes.io~projected/kube-api-access-5hh6m:{mountpoint:/var/lib/kubelet/pods/a04401b8-ddb6-4fed-bac2-2526d011a0ad/volumes/kubernetes.io~projected/kube-api-access-5hh6m major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c/volumes/kubernetes.io~projected/kube-api-access-p6gdd:{mountpoint:/var/lib/kubelet/pods/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c/volumes/kubernetes.io~projected/kube-api-access-p6gdd major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acca86e6-473d-442d-9ca9-8ba949d8caa9/volumes/kubernetes.io~projected/kube-api-access-xgwh8:{mountpoint:/var/lib/kubelet/pods/acca86e6-473d-442d-9ca9-8ba949d8caa9/volumes/kubernetes.io~projected/kube-api-access-xgwh8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2b4d5ca-dead-4cca-b082-91264d877d59/volumes/kubernetes.io~projected/kube-api-access-l2pxb:{mountpoint:/var/lib/kubelet/pods/b2b4d5ca-dead-4cca-b082-91264d877d59/volumes/kubernetes.io~projected/kube-api-access-l2pxb major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2b4d5ca-dead-4cca-b082-91264d877d59/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b2b4d5ca-dead-4cca-b082-91264d877d59/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2165f0f-049f-4588-898f-db7ef275411d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c2165f0f-049f-4588-898f-db7ef275411d/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volumes/kubernetes.io~projected/kube-api-access-s5w6l:{mountpoint:/var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volumes/kubernetes.io~projected/kube-api-access-s5w6l major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5eb73d7-8dd5-4e38-b312-9fd9f4881762/volumes/kubernetes.io~projected/kube-api-access-j4fdl:{mountpoint:/var/lib/kubelet/pods/e5eb73d7-8dd5-4e38-b312-9fd9f4881762/volumes/kubernetes.io~projected/kube-api-access-j4fdl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0705c0f-2c37-412d-a593-76380e0c37e0/volumes/kubernetes.io~projected/kube-api-access-m8jtl:{mountpoint:/var/lib/kubelet/pods/f0705c0f-2c37-412d-a593-76380e0c37e0/volumes/kubernetes.io~projected/kube-api-access-m8jtl major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-101:{mountpoint:/var/lib/containers/storage/overlay/8f4b6370823c654b692760f9436d67fab822f6d205ade5719e581bfdbee82297/merged major:0 minor:101 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/dbf33fa5c7309ab7c8bf1d8bf8c96c567e6babed25628f4d62851a789bc17ad3/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6fa464e3d0f94a7bfe81031f44461a6e4c95f9cabd848d104c965061046235c4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c406377cecda0a41ba531ea7c8fe23d68549b3760556653875506a899f4ce917/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/dfa28fce3cddcd7463040eb3a08ded86f293f454dabf570d7ad95bd7132b651e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a04cf6064fd3aa96b274003b2658da3b3790993a5a23d9c96349d0de10121ae4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d9dbebbe4b60d09c224b2c2c1a6df3001eff249aec3282b865387b87cb7720c0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7da44db25e11ab75a482ae2d6b3858216b99ea1edea01720b710ea381aeff645/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7e1a3f535e5fd8bc3726f6f3d3da83e3cd7a66eb3464d79de5162db8a568522a/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/2f5f35831d838e0da87131b64643f6f10d3d70319e39b16984cede68e98367e5/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/0c2206d041d300c623581144a1dbd79719eea0689355e807197541f92183c2d9/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/e9212a11768dc24001e4e4cdfc1ce3c4b55d7174f76b29ea6db7c7f9e90a83eb/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/5105e605f96e3c7e7230dba311ba3a231e34cedfbdf01d32313d3f1367979847/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c67e7669db6353d35554b2c1b79e3085e2b9fe4ddee85889134d869d366cd1ec/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/71049ad067c0ff3518f8ac47fc79b22d413b09eabc3d8c1d9f56b39a468f6cfa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c8a8bae60a72350ccc01e48bc4517b0ad76bd089a461020623851a4617bc7018/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9ec82eaf03641e163a2c8a8fb96287cf218850a7f99fa008e1c190bf82d4c7f3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/368bd42cb13b89e17df34235ddde513c0831ff392e3a1d8f86a18148c59fdc08/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7d272a4cc5bb9154dbaf515a6827d52a50a7b105a25b277b6ffe771821406edc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/5c9a206cbab9819af31e09021f7c0d34d052a260490b444a4fa4558b55736987/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/e06ffbec7b1f7eaca17e8ed2b518caad99a2305782d803debea1f20d4d494f74/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/4d4a0b3c7cce3adfb8ecc6e4baac0b319f36fecf485c1c28e50e24408d6df457/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/27ec314b1cdb17b41c72eb502c4c019488451af26260f7ed79558ad71a4ab925/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/4ab5d29238963b86539cfbe943494ebd70be49c36852c277169f96594d9685de/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/bf5fd690c35b2d0217d289865945ce5e32d9e3109fa05146ee686bf9cf419833/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/9168df31c6a345ab014f6dfe921a8e7db7db98c41934dd75d3abae8bac03d58e/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/2204763f14427125579fe5bc79ca95b7267586fb422595aa55594a7ad5088be6/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/fcff45380f7726ad6bc8f19361d5a50169af37f14a2c4c138b43f57c20f8b2f9/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/968e19bd78d7577264000ae4a697fd49eb6ecee6baf36bff77d1f648a97abadd/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/ce90c6a6510777ea9e1eb6fa5a4f073313f48c22642a214258990bb7ea9437a5/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/361dec597e060cc96ee8c6725c31d8ec3da5686007fab143243fd40fb625cd90/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/356052446fc5c76d2b773cd4f33626b8887513c8091fa6fb83bfb302eef9b101/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/32f5b23c80bd9e0ea0d9eee9feee27d2a358c60e1c165fabba28a1a8ba863c5b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/f8aeed4d252ce7c1a8e05470a4b14160ba5ecd269fdee685f6de72f498caa728/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/28f891fe26799f94f412b34952111d70ddc7ac75574f783b6cc915ccab49c90d/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c85c9d68343c40a260557cd5a97747bce83f53126a9e1a29407e155cc0c2080c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-58:{mountpoint:/var/lib/containers/storage/overlay/8d9aa171af3a5201300f5cf4578846afca3a53144cfe61cc421bba3eace6f8ce/merged major:0 minor:58 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/431ebbbf9a8409235edbc98b0a864cc812cf8f2d43420a59f720d47976b8b409/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c1de41a0ecf51b347eef46567397940ac4dc7ecf7baae0cee9d58f048fa39927/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d4c565d104beb0020d02b2237322ae72c256a82930a11142b86e5551ebcd23fb/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/89f09ba9d840620a262685e636bdf488265ecbf7f6a97ddd02d2b5b9f0d87136/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/90973e11c4016d2b4e87697682adefcdbf599015cb59fa14383005e819d24ebf/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/827d312f3b9d315981089a71fa1675cf2ea5aab2617031f9596a7863d2174c8a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/70dad6d893b7f206b6b737a39f1e54404cf4f42fb7f61eaae0461275f9c846e0/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/e9f010b703bd5e40d88fc866fd4b90aa55a4ebee085734b4f92100706560cfef/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/66deb5156b8fa29b87ec4d11b1a1c52476885e67c571d00915ed23a431e35593/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 22:12:58.651606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.651199 4625 manager.go:217] Machine: {Timestamp:2026-11-03 22:12:58.650399669 +0000 UTC m=+0.079735657 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115846 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0f8bf6504e2c0edc5b1275b762ed SystemUUID:ec2d0f8b-f650-4e2c-0edc-5b1275b762ed BootID:1db81baa-5719-4853-b841-3fb9639f739a Filesystems:[{Device:/var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d2b8d4b49f6fb1836cbb4f32773ffa180e1ca3963ff3cf0dd239ec45a4abbda/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bd09b8b09d2f4eae5953cd491109d83cf16569fd08c2bf1cf6e428c72c14a47/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~projected/kube-api-access-5lnxp DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f0705c0f-2c37-412d-a593-76380e0c37e0/volumes/kubernetes.io~projected/kube-api-access-m8jtl DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e495d17d943b9577b3777b0f9b4282d657d02b3848e28f99fa5b6673e238741/userdata/shm DeviceMajor:0 DeviceMinor:106 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/73539f96-a0bb-45c0-9ec0-596d7721996c/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6602f77f-4f03-409b-8174-3ae6d73f6420/volumes/kubernetes.io~projected/kube-api-access-gm74n DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c/volumes/kubernetes.io~projected/kube-api-access-p6gdd DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-58 DeviceMajor:0 DeviceMinor:58 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f3e4e964af0a63479dfd41c9f4c4e350dc6e1a862b216a6b1a3555eaa860949/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5eb73d7-8dd5-4e38-b312-9fd9f4881762/volumes/kubernetes.io~projected/kube-api-access-j4fdl DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00c1e219b17813ca93a7a52190c2fd54cc03e406f50c62db1f35c64096b29a44/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eebdb82a457f55b8ffbf3a093e6ebf1efdb444bfcccb2df6f04a8b88c0461a7/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fb1828bfc38c813678aea328f233a195d7ecf8c1ea18ee298404d8a595d604d/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/52a290a0-8456-4ca6-afa3-8324f7c76cc0/volumes/kubernetes.io~projected/kube-api-access-jqkqf DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/941fe33c-1af4-4a8c-bbda-6c3756edae7f/volumes/kubernetes.io~projected/kube-api-access-rjvs5 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/63ada7c0-b6ab-4ba2-bc6d-990b007311a1/volumes/kubernetes.io~projected/kube-api-access-ntqgq DeviceMajor:0 DeviceMinor:68 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cae49f2597fba57d633d2aa1e2c0afb699a58d9cf612dcd55ae1afad6c394aa/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2165f0f-049f-4588-898f-db7ef275411d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volumes/kubernetes.io~projected/kube-api-access-s5w6l DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32c88bd7-b717-4197-b99c-4060840ef9d5/volumes/kubernetes.io~projected/kube-api-access-fpgjk DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acca86e6-473d-442d-9ca9-8ba949d8caa9/volumes/kubernetes.io~projected/kube-api-access-xgwh8 DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73539f96-a0bb-45c0-9ec0-596d7721996c/volumes/kubernetes.io~projected/kube-api-access-dbqkh DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17608ecb-fd95-49bf-a939-b2c1587e4fa1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b2b4d5ca-dead-4cca-b082-91264d877d59/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/548691e3d13dbd44053a2a8716d9dbe685fa1eafa2b3435c26241af81eba8b5a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6294b9cc-3007-4941-bff4-0300e2f6343c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/05f6162d-1577-465d-83c3-b020607dd4e9/volumes/kubernetes.io~projected/kube-api-access-t2wnh DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9156dd68-5a3f-49fd-a004-b736b2dcabce/volumes/kubernetes.io~projected/kube-api-access-9c4fl DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a04401b8-ddb6-4fed-bac2-2526d011a0ad/volumes/kubernetes.io~projected/kube-api-access-5hh6m DeviceMajor:0 DeviceMinor:69 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9156dd68-5a3f-49fd-a004-b736b2dcabce/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b2b4d5ca-dead-4cca-b082-91264d877d59/volumes/kubernetes.io~projected/kube-api-access-l2pxb DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07da56523951ca288bd922e19de9a2a9211ae2ebf88ecd5a0c5abd9bc841dca1/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2630804f37348b7e6d930f4b96f4fdd9d685d00c5f06484f901af01e0cc5015/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae84f6f52b777d1c7fa2233f92e4e6e60729a8b405138decb04d5129b510b0ac/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1d27604a03d0165b240c85c3938698e4958789e1974fabb47b434fcb5ac3155/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bff27fde53144f474e8e4ca5cfe745e5eb2a85a6444c7123a61c1c595d4d6c50/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e55cd659-eeee-4fbf-b11b-c376d306a652/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07da56523951ca2 MacAddress:b6:83:27:ea:e7:75 Speed:10000 Mtu:8901} {Name:0bd09b8b09d2f4e MacAddress:1e:d6:57:5f:6f:a2 Speed:10000 Mtu:8901} {Name:34266ea37bcf416 MacAddress:ca:3c:db:8a:40:ca Speed:10000 Mtu:8901} {Name:6cae49f2597fba5 MacAddress:8e:ac:7a:15:c6:a9 Speed:10000 Mtu:8901} {Name:9d2b8d4b49f6fb1 MacAddress:d2:8c:36:03:a1:84 Speed:10000 Mtu:8901} {Name:ae84f6f52b777d1 MacAddress:ae:a0:55:1a:ea:82 Speed:10000 Mtu:8901} {Name:b1d27604a03d016 MacAddress:3a:d6:01:ca:9d:3a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:ab:fd:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:46:4f:62:6f:42:3c Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:ab:fd:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:97:9f:39:bb:c9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:41:9e:30:09:3f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:12:58.651777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.651612 4625 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:12:58.651777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.651666 4625 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:12:58.651994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.651964 4625 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:12:58.652285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.651995 4625 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:12:58.652285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652127 4625 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:12:58.652285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652135 4625 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:12:58.652285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652145 4625 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:58.652285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652158 4625 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:58.652285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652259 4625 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:12:58.652541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652481 4625 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:12:58.652615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652604 4625 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:12:58.652651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652644 4625 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:12:58.652680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652661 4625 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:12:58.652680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652673 4625 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:12:58.652748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652683 4625 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:12:58.652748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.652693 4625 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:12:58.653360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.653341 4625 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:12:58.653360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.653354 4625 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:58.653946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.653928 4625 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:12:58.666928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.666910 4625 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:12:58.666928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.666926 4625 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:58.667085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667049 4625 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:12:58.667290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667273 4625 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:12:58.667355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667312 4625 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:12:58.667444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667433 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:12:58.667509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667453 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:12:58.667509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667464 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:12:58.667509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667474 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:12:58.667509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667484 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:12:58.667509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667493 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:12:58.667509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667503 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:12:58.667791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667513 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:12:58.667791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667524 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:12:58.667791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667534 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:12:58.667791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667555 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:12:58.667791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667569 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:12:58.667791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667604 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:12:58.667791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.667614 4625 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:12:58.671157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.671139 4625 server.go:1263] "Started kubelet" Nov 03 22:12:58.671698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.671211 4625 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:12:58.671698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.671236 4625 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:12:58.671698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.671328 4625 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:12:58.671675 ip-10-0-128-81 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:12:58.672155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.671769 4625 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:12:58.673779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.673160 4625 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:12:58.677354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.676155 4625 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:12:58.677354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.676999 4625 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:12:58.677354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.677012 4625 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:12:58.677354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.677040 4625 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:12:58.677354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.677158 4625 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:12:58.677354 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:58.677282 4625 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found" Nov 03 22:12:58.678232 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.677883 4625 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:20 +0000 UTC" deadline="2028-07-05 01:36:13.101555726 +0000 UTC" Nov 03 22:12:58.678232 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.678043 4625 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14619h23m14.423518745s" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.678953 4625 factory.go:55] Registering systemd factory Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.678972 4625 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680209 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73539f96-a0bb-45c0-9ec0-596d7721996c" volumeName="kubernetes.io/configmap/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-cabundle" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680241 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acca86e6-473d-442d-9ca9-8ba949d8caa9" volumeName="kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680251 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2165f0f-049f-4588-898f-db7ef275411d" volumeName="kubernetes.io/secret/c2165f0f-049f-4588-898f-db7ef275411d-agent-certs" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680259 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55cd659-eeee-4fbf-b11b-c376d306a652" volumeName="kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680266 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55cd659-eeee-4fbf-b11b-c376d306a652" volumeName="kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-env-overrides" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680273 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941fe33c-1af4-4a8c-bbda-6c3756edae7f" volumeName="kubernetes.io/projected/941fe33c-1af4-4a8c-bbda-6c3756edae7f-kube-api-access-rjvs5" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680281 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32c88bd7-b717-4197-b99c-4060840ef9d5" volumeName="kubernetes.io/configmap/32c88bd7-b717-4197-b99c-4060840ef9d5-config-volume" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680288 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a290a0-8456-4ca6-afa3-8324f7c76cc0" volumeName="kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680297 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a290a0-8456-4ca6-afa3-8324f7c76cc0" volumeName="kubernetes.io/projected/52a290a0-8456-4ca6-afa3-8324f7c76cc0-kube-api-access-jqkqf" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680304 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2b4d5ca-dead-4cca-b082-91264d877d59" volumeName="kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680310 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55cd659-eeee-4fbf-b11b-c376d306a652" volumeName="kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-config" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680326 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5eb73d7-8dd5-4e38-b312-9fd9f4881762" volumeName="kubernetes.io/configmap/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-serviceca" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680333 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0705c0f-2c37-412d-a593-76380e0c37e0" volumeName="kubernetes.io/projected/f0705c0f-2c37-412d-a593-76380e0c37e0-kube-api-access-m8jtl" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680340 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a290a0-8456-4ca6-afa3-8324f7c76cc0" volumeName="kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-binary-copy" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680383 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6294b9cc-3007-4941-bff4-0300e2f6343c" volumeName="kubernetes.io/projected/6294b9cc-3007-4941-bff4-0300e2f6343c-kube-api-access-5lnxp" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680390 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6602f77f-4f03-409b-8174-3ae6d73f6420" volumeName="kubernetes.io/projected/6602f77f-4f03-409b-8174-3ae6d73f6420-kube-api-access-gm74n" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680399 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6ad98a8-5e26-471c-a5a0-0ea41c026e1c" volumeName="kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680406 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acca86e6-473d-442d-9ca9-8ba949d8caa9" volumeName="kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680413 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55cd659-eeee-4fbf-b11b-c376d306a652" volumeName="kubernetes.io/secret/e55cd659-eeee-4fbf-b11b-c376d306a652-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680420 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f6162d-1577-465d-83c3-b020607dd4e9" volumeName="kubernetes.io/projected/05f6162d-1577-465d-83c3-b020607dd4e9-kube-api-access-t2wnh" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680429 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32c88bd7-b717-4197-b99c-4060840ef9d5" volumeName="kubernetes.io/empty-dir/32c88bd7-b717-4197-b99c-4060840ef9d5-tmp-dir" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680437 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ada7c0-b6ab-4ba2-bc6d-990b007311a1" volumeName="kubernetes.io/empty-dir/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-tmp-dir" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680445 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9156dd68-5a3f-49fd-a004-b736b2dcabce" volumeName="kubernetes.io/projected/9156dd68-5a3f-49fd-a004-b736b2dcabce-kube-api-access-9c4fl" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680452 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2b4d5ca-dead-4cca-b082-91264d877d59" volumeName="kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680459 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2b4d5ca-dead-4cca-b082-91264d877d59" volumeName="kubernetes.io/empty-dir/b2b4d5ca-dead-4cca-b082-91264d877d59-data-volume" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680465 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0705c0f-2c37-412d-a593-76380e0c37e0" volumeName="kubernetes.io/configmap/f0705c0f-2c37-412d-a593-76380e0c37e0-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680473 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17608ecb-fd95-49bf-a939-b2c1587e4fa1" volumeName="kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680480 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32c88bd7-b717-4197-b99c-4060840ef9d5" volumeName="kubernetes.io/projected/32c88bd7-b717-4197-b99c-4060840ef9d5-kube-api-access-fpgjk" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680491 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6294b9cc-3007-4941-bff4-0300e2f6343c" volumeName="kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-tuned" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680498 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9156dd68-5a3f-49fd-a004-b736b2dcabce" volumeName="kubernetes.io/secret/9156dd68-5a3f-49fd-a004-b736b2dcabce-cert" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680508 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ada7c0-b6ab-4ba2-bc6d-990b007311a1" volumeName="kubernetes.io/projected/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-kube-api-access-ntqgq" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680515 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a290a0-8456-4ca6-afa3-8324f7c76cc0" volumeName="kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680524 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6294b9cc-3007-4941-bff4-0300e2f6343c" volumeName="kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-tmp" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680531 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73539f96-a0bb-45c0-9ec0-596d7721996c" volumeName="kubernetes.io/projected/73539f96-a0bb-45c0-9ec0-596d7721996c-kube-api-access-dbqkh" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680538 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941fe33c-1af4-4a8c-bbda-6c3756edae7f" volumeName="kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cni-binary-copy" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680546 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2b4d5ca-dead-4cca-b082-91264d877d59" volumeName="kubernetes.io/projected/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-api-access-l2pxb" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680553 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2165f0f-049f-4588-898f-db7ef275411d" volumeName="kubernetes.io/configmap/c2165f0f-049f-4588-898f-db7ef275411d-konnectivity-ca" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680560 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73539f96-a0bb-45c0-9ec0-596d7721996c" volumeName="kubernetes.io/secret/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-key" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680568 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941fe33c-1af4-4a8c-bbda-6c3756edae7f" volumeName="kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-daemon-config" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680575 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a04401b8-ddb6-4fed-bac2-2526d011a0ad" volumeName="kubernetes.io/projected/a04401b8-ddb6-4fed-bac2-2526d011a0ad-kube-api-access-5hh6m" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680583 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55cd659-eeee-4fbf-b11b-c376d306a652" volumeName="kubernetes.io/projected/e55cd659-eeee-4fbf-b11b-c376d306a652-kube-api-access-s5w6l" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680591 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acca86e6-473d-442d-9ca9-8ba949d8caa9" volumeName="kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680598 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acca86e6-473d-442d-9ca9-8ba949d8caa9" volumeName="kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680606 4625 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5eb73d7-8dd5-4e38-b312-9fd9f4881762" volumeName="kubernetes.io/projected/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-kube-api-access-j4fdl" seLinuxMountContext="" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680613 4625 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:12:58.680751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680619 4625 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:12:58.687705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680825 4625 factory.go:153] Registering CRI-O factory Nov 03 22:12:58.687705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680840 4625 factory.go:223] Registration of the crio container factory successfully Nov 03 22:12:58.687705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680914 4625 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:12:58.687705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680937 4625 factory.go:103] Registering Raw factory Nov 03 22:12:58.687705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.680953 4625 manager.go:1199] Started watching for new ooms in manager Nov 03 22:12:58.687705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.681292 4625 manager.go:319] Starting recovery of all containers Nov 03 22:12:58.714105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.714078 4625 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:58.714214 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.714144 4625 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:12:58.718045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.718002 4625 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:12:58.720756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.719443 4625 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:12:58.720756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.719470 4625 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:12:58.720756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.719503 4625 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:12:58.720756 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:58.719550 4625 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:12:58.723367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.723343 4625 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:58.725832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.725805 4625 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:58.734545 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.734520 4625 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:58.745978 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.745947 4625 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="8e7b50840cfded42cd65bf5bc143fb965099fe3424f18e6f07b02a94f7a384f9" exitCode=0 Nov 03 22:12:58.745978 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.745978 4625 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="c34023f39cf3c49e391a65beba1ef4da8801817ad7980d8ae957ee03252964ea" exitCode=0 Nov 03 22:12:58.746093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.745989 4625 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="abe3e95b632faae57a1f5580a376c5f57b3e99366ca67d02719eee2b8fd5962f" exitCode=0 Nov 03 22:12:58.746093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.745998 4625 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="03dec4c5ce14c3f779f0e4deba10dabb990d92568078bafaf7117f51b5b06ed4" exitCode=0 Nov 03 22:12:58.746093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.746007 4625 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="763702e8af47c24996acd88e1680903f78eaa74a9bb19a9d703125024c2048b7" exitCode=0 Nov 03 22:12:58.746093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.746016 4625 generic.go:356] "Generic (PLEG): container finished" podID="52a290a0-8456-4ca6-afa3-8324f7c76cc0" containerID="d61cc0ffd9f8dc3f8ea35397ea6c61eca25ee72a32811de0dff3c693e110d877" exitCode=0 Nov 03 22:12:58.758444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.758386 4625 generic.go:356] "Generic (PLEG): container finished" podID="03c11f74aa32ce1bf6de4a6284c11142" containerID="69627fcb47055014ff62d390b1e64624f05e7b248845faf3f00f6fb54b0b83fd" exitCode=0 Nov 03 22:12:58.767046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.767018 4625 manager.go:324] Recovery completed Nov 03 22:12:58.786263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786240 4625 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:12:58.786263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786257 4625 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:12:58.786429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786278 4625 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:12:58.786429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786398 4625 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:12:58.786429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786409 4625 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:12:58.786429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786427 4625 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:12:58.786616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786435 4625 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:12:58.786616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786441 4625 policy_none.go:50] "Start" Nov 03 22:12:58.786616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786449 4625 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:12:58.786616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.786457 4625 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:12:58.794061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.794041 4625 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:12:58.794061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.794061 4625 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:12:58.794172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.794068 4625 policy_none.go:44] "Start" Nov 03 22:12:58.798601 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.798127 4625 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:12:58.803623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.803609 4625 manager.go:341] "Starting Device Plugin manager" Nov 03 22:12:58.803709 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:58.803633 4625 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:12:58.803709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.803642 4625 server.go:87] "Starting device plugin registration server" Nov 03 22:12:58.803894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.803884 4625 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:12:58.803925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.803894 4625 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:12:58.803997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.803982 4625 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:12:58.804102 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.804076 4625 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:12:58.804151 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.804112 4625 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:12:58.804151 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.804117 4625 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:12:58.804233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.804158 4625 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:12:58.804100328Z","UUID":"a73f5ddc-0490-44b9-b626-19a1d10c0030","Handler":null,"Name":"","Endpoint":""} Nov 03 22:12:58.805497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.805466 4625 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:12:58.805602 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.805515 4625 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:12:58.805602 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:58.805535 4625 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:12:58.819666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.819624 4625 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"] Nov 03 22:12:58.820295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.820257 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"03c11f74aa32ce1bf6de4a6284c11142","Type":"ContainerStarted","Data":"adff9d78183c06f8fcf4262ba722367caca8714f8e6f67eca8a7743c20041a8b"} Nov 03 22:12:58.820432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.820306 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"03c11f74aa32ce1bf6de4a6284c11142","Type":"ContainerDied","Data":"69627fcb47055014ff62d390b1e64624f05e7b248845faf3f00f6fb54b0b83fd"} Nov 03 22:12:58.820432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.820336 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"03c11f74aa32ce1bf6de4a6284c11142","Type":"ContainerStarted","Data":"dd560c533372bb3c1ca19b269b4b7e3428131455bfca08cdbacbc7c6ca899039"} Nov 03 22:12:58.820432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.820375 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" event={"ID":"6cbffddcf197bf46e398ebd8b03a5cd7","Type":"ContainerStarted","Data":"1aac7f42d8c95710572834b11351b688c6137c7777346eba154cfd9f9808e711"} Nov 03 22:12:58.820432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.820406 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" event={"ID":"6cbffddcf197bf46e398ebd8b03a5cd7","Type":"ContainerStarted","Data":"864029dc064b8e4fef5266fe33ea6442216719720022301639a27b6598e0e671"} Nov 03 22:12:58.820550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.820432 4625 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.820686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.820667 4625 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.843178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.843157 4625 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:58.843257 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:58.843197 4625 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.844226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.844211 4625 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:58.844295 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:58.844245 4625 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.904512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.904490 4625 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:58.905641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.905625 4625 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:58.905695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.905662 4625 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:58.905695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.905674 4625 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:58.905783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.905706 4625 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.914841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.914821 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbffddcf197bf46e398ebd8b03a5cd7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"6cbffddcf197bf46e398ebd8b03a5cd7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.914885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.914846 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.914885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.914863 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.948227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.948203 4625 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-81.ec2.internal" Nov 03 22:12:58.948324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:58.948271 4625 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.015204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.015117 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.015204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.015151 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.015204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.015170 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbffddcf197bf46e398ebd8b03a5cd7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"6cbffddcf197bf46e398ebd8b03a5cd7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.015409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.015231 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.015409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.015232 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c11f74aa32ce1bf6de4a6284c11142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"03c11f74aa32ce1bf6de4a6284c11142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.015409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.015227 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbffddcf197bf46e398ebd8b03a5cd7-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"6cbffddcf197bf46e398ebd8b03a5cd7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.653173 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.653143 4625 apiserver.go:52] "Watching apiserver" Nov 03 22:12:59.701240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.701206 4625 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:12:59.701750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.701689 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pwbkp","openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw","openshift-network-operator/iptables-alerter-f9gq2","openshift-cluster-node-tuning-operator/tuned-c26j6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6","kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal","openshift-insights/insights-runtime-extractor-wkl88","openshift-ingress-canary/ingress-canary-9dqrx","openshift-ovn-kubernetes/ovnkube-node-49k6b","kube-system/global-pull-secret-syncer-nvbcj","openshift-dns/dns-default-4hrjm","openshift-image-registry/node-ca-g7nfl","openshift-insights/periodic-gathering-q8k65-h5z45","openshift-multus/multus-additional-cni-plugins-qphlr","kube-system/konnectivity-agent-qwdnc","openshift-dns/node-resolver-97blh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal","openshift-multus/multus-4t752","openshift-network-diagnostics/network-check-target-xpshj","openshift-service-ca/service-ca-5fffc97bdf-fztdk"] Nov 03 22:12:59.703502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.703478 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:59.708919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.708896 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.717625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717589 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mq5fv\"" Nov 03 22:12:59.717625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717617 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tpvw\"" Nov 03 22:12:59.717794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717663 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:12:59.717794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717748 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wvrhf\"" Nov 03 22:12:59.717903 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717811 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.717903 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717829 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7shc8\"" Nov 03 22:12:59.717903 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717823 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.718020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717922 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:12:59.718020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717924 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.718020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717921 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.718020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.717934 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:12:59.718020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718019 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.718247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718018 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:12:59.718247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718055 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.718247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718062 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:12:59.718247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718104 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.718247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718141 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.718859 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718844 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.718959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718876 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:12:59.718959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718881 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.718959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.718908 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz46p\"" Nov 03 22:12:59.719185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.719092 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zdncz\"" Nov 03 22:12:59.719479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.719465 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:12:59.727706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.727686 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwdqh\"" Nov 03 22:12:59.728165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.728133 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4jdd\"" Nov 03 22:12:59.728233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.728220 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nt8s4\"" Nov 03 22:12:59.728874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.728859 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:12:59.728957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.728944 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:12:59.729079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.729066 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wf9zk\"" Nov 03 22:12:59.729121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.729114 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d79qx\"" Nov 03 22:12:59.729647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.729636 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:12:59.730637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730623 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:12:59.730798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730783 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.730877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730861 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 22:12:59.730930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730896 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:12:59.730930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730899 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:12:59.730930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730910 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.730930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730915 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.730930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730930 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.731131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.730958 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.731191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.731158 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.731246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.731165 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.731246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.731181 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.731246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.731185 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.731350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.731259 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:12:59.731350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.731288 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.733466 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.733452 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:12:59.734000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.733986 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 22:12:59.734242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734227 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwzw5\"" Nov 03 22:12:59.734401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734388 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svbdj\"" Nov 03 22:12:59.734401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734397 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:12:59.734522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734402 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdchn\"" Nov 03 22:12:59.734522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734430 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6vzq\"" Nov 03 22:12:59.734522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734496 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:12:59.734522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734499 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjknz\"" Nov 03 22:12:59.734648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734526 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9wkj9\"" Nov 03 22:12:59.734648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.734556 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9h6fz\"" Nov 03 22:12:59.776478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.776458 4625 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.776588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.776575 4625 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.778569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.778555 4625 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:12:59.797980 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.797962 4625 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:59.798051 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:59.797997 4625 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.798485 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.798472 4625 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:59.798525 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:59.798504 4625 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" Nov 03 22:12:59.817922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.817906 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.818007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.817959 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:12:59.818007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.817971 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.818007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.817980 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 22:12:59.818007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.817998 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:12:59.818223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818030 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:12:59.818223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818032 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:12:59.818223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818033 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 22:12:59.818223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818154 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-device-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.818223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818176 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-conf-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.818223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818190 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cnibin\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.818223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818206 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.818521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818225 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.818521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818300 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkqf\" (UniqueName: \"kubernetes.io/projected/52a290a0-8456-4ca6-afa3-8324f7c76cc0-kube-api-access-jqkqf\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.818521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818391 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-key\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:59.818652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818574 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-cabundle\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:59.818652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818603 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.818760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818684 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.818818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818800 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-systemd\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.818957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818935 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-registration-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.819051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818957 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-cabundle\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:59.819051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818980 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wnh\" (UniqueName: \"kubernetes.io/projected/05f6162d-1577-465d-83c3-b020607dd4e9-kube-api-access-t2wnh\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.819051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.818988 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/73539f96-a0bb-45c0-9ec0-596d7721996c-signing-key\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:59.819177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819050 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-socket-dir-parent\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.819177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819078 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hh6m\" (UniqueName: \"kubernetes.io/projected/a04401b8-ddb6-4fed-bac2-2526d011a0ad-kube-api-access-5hh6m\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:59.819177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819098 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbqkh\" (UniqueName: \"kubernetes.io/projected/73539f96-a0bb-45c0-9ec0-596d7721996c-kube-api-access-dbqkh\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:59.819177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819118 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0705c0f-2c37-412d-a593-76380e0c37e0-iptables-alerter-script\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:59.819349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819202 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jtl\" (UniqueName: \"kubernetes.io/projected/f0705c0f-2c37-412d-a593-76380e0c37e0-kube-api-access-m8jtl\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:59.819349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819253 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgjk\" (UniqueName: \"kubernetes.io/projected/32c88bd7-b717-4197-b99c-4060840ef9d5-kube-api-access-fpgjk\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.819349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819277 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-lib-modules\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.819483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819462 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0705c0f-2c37-412d-a593-76380e0c37e0-iptables-alerter-script\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:59.819585 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819526 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-socket-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.819658 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819624 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-sys-fs\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.819730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819690 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.819781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819739 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-k8s-cni-cncf-io\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.819849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819825 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.819912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819892 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-conf\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.819944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819928 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-kubelet\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.819981 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819951 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-systemd-units\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.820032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.819976 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-var-lib-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.820079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820021 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.820126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820086 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-kubelet-config\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.820229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820197 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-run\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.820380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820247 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.820380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820277 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-etc-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.820380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820345 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-system-cni-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.820490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820388 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-tmp-dir\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:59.820490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820420 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-kube-api-access-ntqgq\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:59.820490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820442 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4fl\" (UniqueName: \"kubernetes.io/projected/9156dd68-5a3f-49fd-a004-b736b2dcabce-kube-api-access-9c4fl\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:59.820490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820464 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.820650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820497 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.820650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820501 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-tmp-dir\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:59.820650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820515 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2165f0f-049f-4588-898f-db7ef275411d-konnectivity-ca\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:59.820650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820535 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cni-binary-copy\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820916 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-dbus\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.820993 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9156dd68-5a3f-49fd-a004-b736b2dcabce-cert\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821043 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2b4d5ca-dead-4cca-b082-91264d877d59-crio-socket\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821103 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b4d5ca-dead-4cca-b082-91264d877d59-data-volume\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821175 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pxb\" (UniqueName: \"kubernetes.io/projected/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-api-access-l2pxb\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821187 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2165f0f-049f-4588-898f-db7ef275411d-konnectivity-ca\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821228 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821260 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-host\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.821376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821339 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b4d5ca-dead-4cca-b082-91264d877d59-data-volume\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.821864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821662 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cnibin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.821908 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822084 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-hostroot\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822125 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822148 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9156dd68-5a3f-49fd-a004-b736b2dcabce-cert\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822165 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822208 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-tmp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822244 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-system-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822262 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822277 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-netns\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822315 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm74n\" (UniqueName: \"kubernetes.io/projected/6602f77f-4f03-409b-8174-3ae6d73f6420-kube-api-access-gm74n\") pod \"migrator-66df5d496b-6mxpw\" (UID: \"6602f77f-4f03-409b-8174-3ae6d73f6420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822357 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822396 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-log-socket\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822429 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5w6l\" (UniqueName: \"kubernetes.io/projected/e55cd659-eeee-4fbf-b11b-c376d306a652-kube-api-access-s5w6l\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822530 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822576 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwh8\" (UniqueName: \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822613 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32c88bd7-b717-4197-b99c-4060840ef9d5-config-volume\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822637 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-bin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822689 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-multus\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.822899 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17608ecb-fd95-49bf-a939-b2c1587e4fa1-original-pull-secret\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823110 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-daemon-config\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823207 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-kubelet\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823211 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823241 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-bin\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823318 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32c88bd7-b717-4197-b99c-4060840ef9d5-config-volume\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823468 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-config\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823509 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823548 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-modprobe-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823571 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823763 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-var-lib-kubelet\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823788 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-daemon-config\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823850 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-tmp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823930 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-multus-certs\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823953 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-slash\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.823999 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-script-lib\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824042 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-host\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824104 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-config\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824158 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-kubernetes\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824170 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cni-binary-copy\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824189 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-sys\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824233 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824275 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-ovnkube-script-lib\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824289 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-etc-kubernetes\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824341 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-netns\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824378 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824420 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-env-overrides\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824448 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-tuned\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824508 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-tuned\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824548 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lnxp\" (UniqueName: \"kubernetes.io/projected/6294b9cc-3007-4941-bff4-0300e2f6343c-kube-api-access-5lnxp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824584 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvs5\" (UniqueName: \"kubernetes.io/projected/941fe33c-1af4-4a8c-bbda-6c3756edae7f-kube-api-access-rjvs5\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824614 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-netd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824649 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55cd659-eeee-4fbf-b11b-c376d306a652-ovn-node-metrics-cert\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824670 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824676 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e55cd659-eeee-4fbf-b11b-c376d306a652-env-overrides\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824694 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-hosts-file\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824754 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4fdl\" (UniqueName: \"kubernetes.io/projected/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-kube-api-access-j4fdl\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824775 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c88bd7-b717-4197-b99c-4060840ef9d5-tmp-dir\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824796 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysconfig\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824818 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-systemd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824846 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-ovn\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824911 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-node-log\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824932 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-serviceca\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824956 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.824977 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2165f0f-049f-4588-898f-db7ef275411d-agent-certs\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.825209 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2165f0f-049f-4588-898f-db7ef275411d-agent-certs\") pod \"konnectivity-agent-qwdnc\" (UID: \"c2165f0f-049f-4588-898f-db7ef275411d\") " pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.825495 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55cd659-eeee-4fbf-b11b-c376d306a652-ovn-node-metrics-cert\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.825573 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c88bd7-b717-4197-b99c-4060840ef9d5-tmp-dir\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.825613 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0705c0f-2c37-412d-a593-76380e0c37e0-host-slash\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.825777 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.825819 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.825838 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-serviceca\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:59.826191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.826107 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.828198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.826469 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2b4d5ca-dead-4cca-b082-91264d877d59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.831785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.831768 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.838538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.838520 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkqf\" (UniqueName: \"kubernetes.io/projected/52a290a0-8456-4ca6-afa3-8324f7c76cc0-kube-api-access-jqkqf\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.839148 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.839130 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wnh\" (UniqueName: \"kubernetes.io/projected/05f6162d-1577-465d-83c3-b020607dd4e9-kube-api-access-t2wnh\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.840237 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.840213 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgjk\" (UniqueName: \"kubernetes.io/projected/32c88bd7-b717-4197-b99c-4060840ef9d5-kube-api-access-fpgjk\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.840379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.840344 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jtl\" (UniqueName: \"kubernetes.io/projected/f0705c0f-2c37-412d-a593-76380e0c37e0-kube-api-access-m8jtl\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:59.840430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.840343 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hh6m\" (UniqueName: \"kubernetes.io/projected/a04401b8-ddb6-4fed-bac2-2526d011a0ad-kube-api-access-5hh6m\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:59.840599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.840584 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbqkh\" (UniqueName: \"kubernetes.io/projected/73539f96-a0bb-45c0-9ec0-596d7721996c-kube-api-access-dbqkh\") pod \"service-ca-5fffc97bdf-fztdk\" (UID: \"73539f96-a0bb-45c0-9ec0-596d7721996c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fztdk" Nov 03 22:12:59.841077 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.841045 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pxb\" (UniqueName: \"kubernetes.io/projected/b2b4d5ca-dead-4cca-b082-91264d877d59-kube-api-access-l2pxb\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.841173 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.841155 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4fl\" (UniqueName: \"kubernetes.io/projected/9156dd68-5a3f-49fd-a004-b736b2dcabce-kube-api-access-9c4fl\") pod \"ingress-canary-9dqrx\" (UID: \"9156dd68-5a3f-49fd-a004-b736b2dcabce\") " pod="openshift-ingress-canary/ingress-canary-9dqrx" Nov 03 22:12:59.841361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.841342 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-kube-api-access-ntqgq\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:59.845812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.845794 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm74n\" (UniqueName: \"kubernetes.io/projected/6602f77f-4f03-409b-8174-3ae6d73f6420-kube-api-access-gm74n\") pod \"migrator-66df5d496b-6mxpw\" (UID: \"6602f77f-4f03-409b-8174-3ae6d73f6420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6mxpw" Nov 03 22:12:59.846016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.846000 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6gdd\" (UniqueName: \"kubernetes.io/projected/a6ad98a8-5e26-471c-a5a0-0ea41c026e1c-kube-api-access-p6gdd\") pod \"network-check-target-xpshj\" (UID: \"a6ad98a8-5e26-471c-a5a0-0ea41c026e1c\") " pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:12:59.846063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.846004 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5w6l\" (UniqueName: \"kubernetes.io/projected/e55cd659-eeee-4fbf-b11b-c376d306a652-kube-api-access-s5w6l\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.846932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.846913 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwh8\" (UniqueName: \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\") pod \"periodic-gathering-q8k65-h5z45\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:12:59.847108 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.847086 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvs5\" (UniqueName: \"kubernetes.io/projected/941fe33c-1af4-4a8c-bbda-6c3756edae7f-kube-api-access-rjvs5\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.847178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.847156 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4fdl\" (UniqueName: \"kubernetes.io/projected/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-kube-api-access-j4fdl\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:59.847211 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.847201 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lnxp\" (UniqueName: \"kubernetes.io/projected/6294b9cc-3007-4941-bff4-0300e2f6343c-kube-api-access-5lnxp\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926089 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-kubelet\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926130 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-bin\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926150 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-modprobe-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926172 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-var-lib-kubelet\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926206 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-kubelet\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926229 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-bin\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926250 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-multus-certs\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926234 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-var-lib-kubelet\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926269 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-slash\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926287 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-host\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926294 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-multus-certs\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926308 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-kubernetes\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926310 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-modprobe-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926322 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-sys\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926350 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-sys\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926339 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5eb73d7-8dd5-4e38-b312-9fd9f4881762-host\") pod \"node-ca-g7nfl\" (UID: \"e5eb73d7-8dd5-4e38-b312-9fd9f4881762\") " pod="openshift-image-registry/node-ca-g7nfl" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926367 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-slash\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926380 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-kubernetes\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926378 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926402 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926415 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-etc-kubernetes\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926438 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-netns\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926460 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-etc-kubernetes\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926480 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926489 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-netns\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926508 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-netd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.926506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926513 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926538 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926548 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-cni-netd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926561 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-hosts-file\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926584 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysconfig\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926610 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-systemd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926628 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-ovn\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926641 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-node-log\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926661 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0705c0f-2c37-412d-a593-76380e0c37e0-host-slash\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926677 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63ada7c0-b6ab-4ba2-bc6d-990b007311a1-hosts-file\") pod \"node-resolver-97blh\" (UID: \"63ada7c0-b6ab-4ba2-bc6d-990b007311a1\") " pod="openshift-dns/node-resolver-97blh" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926682 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926631 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysconfig\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926692 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-ovn\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926708 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-device-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926737 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-node-log\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926737 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0705c0f-2c37-412d-a593-76380e0c37e0-host-slash\") pod \"iptables-alerter-f9gq2\" (UID: \"f0705c0f-2c37-412d-a593-76380e0c37e0\") " pod="openshift-network-operator/iptables-alerter-f9gq2" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926772 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-conf-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926776 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926801 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-device-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926802 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cnibin\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926834 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-conf-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926845 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-cnibin\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926847 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-systemd\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926882 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-run-systemd\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926889 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-systemd\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926892 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-registration-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926919 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-socket-dir-parent\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926945 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-lib-modules\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926951 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-registration-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926966 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-socket-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926957 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-socket-dir-parent\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926953 4625 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.926989 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-sys-fs\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927001 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-lib-modules\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927011 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927028 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-k8s-cni-cncf-io\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927037 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-socket-dir\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927042 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-sys-fs\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927042 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927072 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-k8s-cni-cncf-io\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927082 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-multus-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927087 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-conf\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927114 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-kubelet\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927129 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-systemd-units\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927167 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-var-lib-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927171 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-d\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927194 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927200 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-kubelet\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927206 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-etc-sysctl-conf\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927194 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-systemd-units\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927219 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-kubelet-config\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927222 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-var-lib-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927242 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-run\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927257 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-host-run-ovn-kubernetes\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927260 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-kubelet-config\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927269 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927311 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-run\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927322 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-etc-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927320 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05f6162d-1577-465d-83c3-b020607dd4e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-k89w6\" (UID: \"05f6162d-1577-465d-83c3-b020607dd4e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k89w6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927341 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-system-cni-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927361 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-system-cni-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927364 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-dbus\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927380 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2b4d5ca-dead-4cca-b082-91264d877d59-crio-socket\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927386 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-etc-openvswitch\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927433 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17608ecb-fd95-49bf-a939-b2c1587e4fa1-dbus\") pod \"global-pull-secret-syncer-nvbcj\" (UID: \"17608ecb-fd95-49bf-a939-b2c1587e4fa1\") " pod="kube-system/global-pull-secret-syncer-nvbcj" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927433 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2b4d5ca-dead-4cca-b082-91264d877d59-crio-socket\") pod \"insights-runtime-extractor-wkl88\" (UID: \"b2b4d5ca-dead-4cca-b082-91264d877d59\") " pod="openshift-insights/insights-runtime-extractor-wkl88" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927452 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-host\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927473 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cnibin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927486 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-hostroot\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927501 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927519 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927543 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-system-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927556 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-cnibin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927565 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-netns\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.927524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927574 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a290a0-8456-4ca6-afa3-8324f7c76cc0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qphlr\" (UID: \"52a290a0-8456-4ca6-afa3-8324f7c76cc0\") " pod="openshift-multus/multus-additional-cni-plugins-qphlr" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927566 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-hostroot\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927593 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-log-socket\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927610 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-run-netns\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927616 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-system-cni-dir\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927627 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e55cd659-eeee-4fbf-b11b-c376d306a652-log-socket\") pod \"ovnkube-node-49k6b\" (UID: \"e55cd659-eeee-4fbf-b11b-c376d306a652\") " pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927593 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6294b9cc-3007-4941-bff4-0300e2f6343c-host\") pod \"tuned-c26j6\" (UID: \"6294b9cc-3007-4941-bff4-0300e2f6343c\") " pod="openshift-cluster-node-tuning-operator/tuned-c26j6" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927621 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-bin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927648 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-bin\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927663 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-multus\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.927730 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941fe33c-1af4-4a8c-bbda-6c3756edae7f-host-var-lib-cni-multus\") pod \"multus-4t752\" (UID: \"941fe33c-1af4-4a8c-bbda-6c3756edae7f\") " pod="openshift-multus/multus-4t752" Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:59.927708 4625 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:12:59.927818 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:00.427801814 +0000 UTC m=+1.857137798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:12:59.930860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:12:59.929113 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04401b8-ddb6-4fed-bac2-2526d011a0ad-metrics-certs\") pod \"network-metrics-daemon-pwbkp\" (UID: \"a04401b8-ddb6-4fed-bac2-2526d011a0ad\") " pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:13:00.003901 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:00.003798 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwbkp" Nov 03 22:13:00.010875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:00.010847 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:13:00.332911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:00.332839 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pwbkp"] Nov 03 22:13:00.342195 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:00.342149 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04401b8_ddb6_4fed_bac2_2526d011a0ad.slice/crio-c771c3603ddf00881e3ca4a367e5a57c4253f3b342075bbe5c85cbfb54b2c329 WatchSource:0}: Error finding container c771c3603ddf00881e3ca4a367e5a57c4253f3b342075bbe5c85cbfb54b2c329: Status 404 returned error can't find the container with id c771c3603ddf00881e3ca4a367e5a57c4253f3b342075bbe5c85cbfb54b2c329 Nov 03 22:13:00.432029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:00.431983 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:00.432171 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:00.432134 4625 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:13:00.432225 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:00.432194 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:01.432180276 +0000 UTC m=+2.861516250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:13:00.781756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:00.781670 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pwbkp" event={"ID":"a04401b8-ddb6-4fed-bac2-2526d011a0ad","Type":"ContainerStarted","Data":"c771c3603ddf00881e3ca4a367e5a57c4253f3b342075bbe5c85cbfb54b2c329"} Nov 03 22:13:01.332320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:01.332281 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:13:01.441933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:01.441895 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:01.442108 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:01.442082 4625 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:13:01.442217 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:01.442202 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:03.442178025 +0000 UTC m=+4.871514016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:13:01.787295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:01.787265 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkl88" event={"ID":"b2b4d5ca-dead-4cca-b082-91264d877d59","Type":"ContainerStarted","Data":"f1ba809d9482d29f3204f40924c248766d504cafd9bf8708e3de77adbec19e4f"} Nov 03 22:13:01.788962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:01.788941 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pwbkp" event={"ID":"a04401b8-ddb6-4fed-bac2-2526d011a0ad","Type":"ContainerStarted","Data":"15741005e3ef6d84e1885561cf2c93a7f126734fa90c512485084fa9517b76a4"} Nov 03 22:13:01.789090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:01.788984 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pwbkp" event={"ID":"a04401b8-ddb6-4fed-bac2-2526d011a0ad","Type":"ContainerStarted","Data":"c1388e18639c01d5e3b305c735928a6d671b19275d867d1a8d3d54add2004cbe"} Nov 03 22:13:02.183324 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:02.183290 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:13:03.455460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:03.455426 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:03.455890 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:03.455552 4625 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:13:03.455890 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:03.455614 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls podName:32c88bd7-b717-4197-b99c-4060840ef9d5 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:07.455600015 +0000 UTC m=+8.884935989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls") pod "dns-default-4hrjm" (UID: "32c88bd7-b717-4197-b99c-4060840ef9d5") : secret "dns-default-metrics-tls" not found Nov 03 22:13:07.479563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:07.479522 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:13:07.479985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:07.479597 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:07.481870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:07.481846 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32c88bd7-b717-4197-b99c-4060840ef9d5-metrics-tls\") pod \"dns-default-4hrjm\" (UID: \"32c88bd7-b717-4197-b99c-4060840ef9d5\") " pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:07.509682 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:07.509662 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:07.805305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:07.805227 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4hrjm"] Nov 03 22:13:07.810043 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:07.810016 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c88bd7_b717_4197_b99c_4060840ef9d5.slice/crio-8a89393437034f19d9663dc2c1e57e9a7ff510f8f8c589889bcb73ce3fb44453 WatchSource:0}: Error finding container 8a89393437034f19d9663dc2c1e57e9a7ff510f8f8c589889bcb73ce3fb44453: Status 404 returned error can't find the container with id 8a89393437034f19d9663dc2c1e57e9a7ff510f8f8c589889bcb73ce3fb44453 Nov 03 22:13:08.811916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:08.811882 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4hrjm" event={"ID":"32c88bd7-b717-4197-b99c-4060840ef9d5","Type":"ContainerStarted","Data":"8a89393437034f19d9663dc2c1e57e9a7ff510f8f8c589889bcb73ce3fb44453"} Nov 03 22:13:09.032998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.032963 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:13:09.033207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.033190 4625 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:13:09.419667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.419635 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lmzk9-8dvsz"] Nov 03 22:13:09.422492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.422471 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.445218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.445185 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lmzk9-8dvsz"] Nov 03 22:13:09.491181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.491151 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58c6431b-8263-435c-8461-74365d4f9ce6-archives-path\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.491181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.491187 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-trusted-ca-bundle\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.491389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.491211 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpj8c\" (UniqueName: \"kubernetes.io/projected/58c6431b-8263-435c-8461-74365d4f9ce6-kube-api-access-hpj8c\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.491389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.491236 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-service-ca-bundle\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.542843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.542767 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f7df997d-hgx6n"] Nov 03 22:13:09.544778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.544760 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.564209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.564184 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 22:13:09.564333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.564256 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 22:13:09.564333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.564279 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:13:09.564333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.564280 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 22:13:09.564434 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.564380 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:13:09.567541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.567521 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f7df997d-hgx6n"] Nov 03 22:13:09.570107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.570084 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 22:13:09.570214 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.570123 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv4dg\"" Nov 03 22:13:09.570214 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.570180 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 22:13:09.591916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.591898 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58c6431b-8263-435c-8461-74365d4f9ce6-archives-path\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.592019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.591925 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-trusted-ca-bundle\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.592019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.591951 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpj8c\" (UniqueName: \"kubernetes.io/projected/58c6431b-8263-435c-8461-74365d4f9ce6-kube-api-access-hpj8c\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.592019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.591993 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-service-ca-bundle\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.592343 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.592324 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58c6431b-8263-435c-8461-74365d4f9ce6-archives-path\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.592793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.592774 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-service-ca-bundle\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.593008 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.592993 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-trusted-ca-bundle\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.613942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.613916 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpj8c\" (UniqueName: \"kubernetes.io/projected/58c6431b-8263-435c-8461-74365d4f9ce6-kube-api-access-hpj8c\") pod \"periodic-gathering-lmzk9-8dvsz\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.692260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.692224 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-service-ca\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.692431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.692265 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-oauth-config\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.692431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.692288 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-config\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.692431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.692400 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2xr\" (UniqueName: \"kubernetes.io/projected/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-kube-api-access-hw2xr\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.692598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.692432 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-oauth-serving-cert\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.692598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.692450 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-serving-cert\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.692598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.692470 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-tmp\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.757332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.757301 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:13:09.793919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.793847 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-oauth-serving-cert\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.793919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.793892 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-serving-cert\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.793923 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-tmp\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.793988 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-service-ca\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.794017 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-oauth-config\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.794050 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-config\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.794092 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2xr\" (UniqueName: \"kubernetes.io/projected/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-kube-api-access-hw2xr\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.794363 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-tmp\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.794691 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-oauth-serving-cert\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.794812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.794706 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-service-ca\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.795051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.795029 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-config\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.796535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.796517 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-oauth-config\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.796634 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.796618 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-serving-cert\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.816900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.816876 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4hrjm" event={"ID":"32c88bd7-b717-4197-b99c-4060840ef9d5","Type":"ContainerStarted","Data":"8f7251bf5543e6f3a9dc42965fa87277bdc05db0113960ff94156b7c350e49dc"} Nov 03 22:13:09.817294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.816926 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4hrjm" event={"ID":"32c88bd7-b717-4197-b99c-4060840ef9d5","Type":"ContainerStarted","Data":"455e5bf1b10d28419b2cac746858e4da8a8e0266a3c9910bcf937e8e4de26e32"} Nov 03 22:13:09.817294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.816968 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:09.840599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.840577 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2xr\" (UniqueName: \"kubernetes.io/projected/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-kube-api-access-hw2xr\") pod \"console-79f7df997d-hgx6n\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:09.854775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:09.854756 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:10.051768 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:10.051691 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lmzk9-8dvsz"] Nov 03 22:13:10.056386 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:10.056357 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c6431b_8263_435c_8461_74365d4f9ce6.slice/crio-7ab5a7b615b98a8c418babf0b2a381fb4c9f04a25f5c96f65846a173f1840fbf WatchSource:0}: Error finding container 7ab5a7b615b98a8c418babf0b2a381fb4c9f04a25f5c96f65846a173f1840fbf: Status 404 returned error can't find the container with id 7ab5a7b615b98a8c418babf0b2a381fb4c9f04a25f5c96f65846a173f1840fbf Nov 03 22:13:10.153871 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:10.153841 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f7df997d-hgx6n"] Nov 03 22:13:10.158597 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:10.158567 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6dbb8a3_be8d_4560_9d4f_b6223ff7ab32.slice/crio-6e39bea3612e8933c0e56b52d6e403fd0889d52ff4528d4162627f6982234289 WatchSource:0}: Error finding container 6e39bea3612e8933c0e56b52d6e403fd0889d52ff4528d4162627f6982234289: Status 404 returned error can't find the container with id 6e39bea3612e8933c0e56b52d6e403fd0889d52ff4528d4162627f6982234289 Nov 03 22:13:10.824565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:10.824526 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" event={"ID":"58c6431b-8263-435c-8461-74365d4f9ce6","Type":"ContainerStarted","Data":"d8a3f2988d239f3e10d0174d5e4bd60adcf196f6fccef5839b096235e133fa25"} Nov 03 22:13:10.825025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:10.824619 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" event={"ID":"58c6431b-8263-435c-8461-74365d4f9ce6","Type":"ContainerStarted","Data":"7ab5a7b615b98a8c418babf0b2a381fb4c9f04a25f5c96f65846a173f1840fbf"} Nov 03 22:13:10.825844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:10.825803 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f7df997d-hgx6n" event={"ID":"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32","Type":"ContainerStarted","Data":"6e39bea3612e8933c0e56b52d6e403fd0889d52ff4528d4162627f6982234289"} Nov 03 22:13:10.862875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:10.862824 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" podStartSLOduration=1.862811666 podStartE2EDuration="1.862811666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:10.862288601 +0000 UTC m=+12.291624595" watchObservedRunningTime="2026-11-03 22:13:10.862811666 +0000 UTC m=+12.292147658" Nov 03 22:13:11.362277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:11.362247 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4hrjm_32c88bd7-b717-4197-b99c-4060840ef9d5/dns/0.log" Nov 03 22:13:11.497751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:11.497706 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4hrjm_32c88bd7-b717-4197-b99c-4060840ef9d5/kube-rbac-proxy/0.log" Nov 03 22:13:12.696257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:12.696228 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97blh_63ada7c0-b6ab-4ba2-bc6d-990b007311a1/dns-node-resolver/0.log" Nov 03 22:13:13.837539 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:13.837501 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f7df997d-hgx6n" event={"ID":"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32","Type":"ContainerStarted","Data":"edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe"} Nov 03 22:13:13.878024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:13.877967 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79f7df997d-hgx6n" podStartSLOduration=2.000267901 podStartE2EDuration="4.877949405s" totalImagesPullingTime="2.877681504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:13.877107005 +0000 UTC m=+15.306442998" watchObservedRunningTime="2026-11-03 22:13:13.877949405 +0000 UTC m=+15.307285400" Nov 03 22:13:13.896058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:13.896031 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g7nfl_e5eb73d7-8dd5-4e38-b312-9fd9f4881762/node-ca/0.log" Nov 03 22:13:14.404557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.404524 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d49ff4b4c-8f5hj"] Nov 03 22:13:14.407179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.407159 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.455399 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.455363 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 22:13:14.456085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.456066 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d49ff4b4c-8f5hj"] Nov 03 22:13:14.497091 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.497046 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9dqrx_9156dd68-5a3f-49fd-a004-b736b2dcabce/serve-healthcheck-canary/0.log" Nov 03 22:13:14.538293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538250 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-trusted-ca-bundle\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.538293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538293 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-oauth-serving-cert\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.538476 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538384 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-console-config\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.538476 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538450 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-service-ca\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.538546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538477 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-serving-cert\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.538546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538509 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x6jx\" (UniqueName: \"kubernetes.io/projected/428eb929-6f4b-41b9-8452-606272ed7650-kube-api-access-4x6jx\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.538546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538529 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-oauth-config\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.538638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.538545 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428eb929-6f4b-41b9-8452-606272ed7650-tmp\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.639914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.639877 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-service-ca\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.640734 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.640696 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-service-ca\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.640856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.640780 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-serving-cert\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.640856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.640821 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x6jx\" (UniqueName: \"kubernetes.io/projected/428eb929-6f4b-41b9-8452-606272ed7650-kube-api-access-4x6jx\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.640948 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.640858 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-oauth-config\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.640948 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.640884 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428eb929-6f4b-41b9-8452-606272ed7650-tmp\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.640948 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.640924 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-trusted-ca-bundle\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.641076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.640953 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-oauth-serving-cert\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.641076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.641018 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-console-config\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.641626 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.641601 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-console-config\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.644159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.644108 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428eb929-6f4b-41b9-8452-606272ed7650-tmp\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.644626 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.644572 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-oauth-serving-cert\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.644844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.644825 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-serving-cert\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.644917 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.644874 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-trusted-ca-bundle\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.646188 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.646167 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-oauth-config\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.666010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.665947 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x6jx\" (UniqueName: \"kubernetes.io/projected/428eb929-6f4b-41b9-8452-606272ed7650-kube-api-access-4x6jx\") pod \"console-d49ff4b4c-8f5hj\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:14.717014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:14.716983 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:15.023572 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:15.023490 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d49ff4b4c-8f5hj"] Nov 03 22:13:15.027871 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:15.027841 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428eb929_6f4b_41b9_8452_606272ed7650.slice/crio-c2c4c12882f3efcca9ba0bd717e6184010e2b5f7301a49f8a3e97bd617d4ac6c WatchSource:0}: Error finding container c2c4c12882f3efcca9ba0bd717e6184010e2b5f7301a49f8a3e97bd617d4ac6c: Status 404 returned error can't find the container with id c2c4c12882f3efcca9ba0bd717e6184010e2b5f7301a49f8a3e97bd617d4ac6c Nov 03 22:13:15.845345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:15.845312 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d49ff4b4c-8f5hj" event={"ID":"428eb929-6f4b-41b9-8452-606272ed7650","Type":"ContainerStarted","Data":"5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951"} Nov 03 22:13:15.845488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:15.845364 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d49ff4b4c-8f5hj" event={"ID":"428eb929-6f4b-41b9-8452-606272ed7650","Type":"ContainerStarted","Data":"c2c4c12882f3efcca9ba0bd717e6184010e2b5f7301a49f8a3e97bd617d4ac6c"} Nov 03 22:13:15.888389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:15.888339 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d49ff4b4c-8f5hj" podStartSLOduration=1.8883270699999999 podStartE2EDuration="1.88832707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:15.887510496 +0000 UTC m=+17.316846488" watchObservedRunningTime="2026-11-03 22:13:15.88832707 +0000 UTC m=+17.317663054" Nov 03 22:13:16.008605 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.008572 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwdnc" Nov 03 22:13:16.334997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.334922 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5"] Nov 03 22:13:16.337280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.337255 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.359512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.359485 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5"] Nov 03 22:13:16.360677 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.360652 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:13:16.369375 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.369353 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:13:16.369517 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.369381 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 22:13:16.369630 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.369616 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:13:16.369691 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.369683 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:13:16.369771 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.369757 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bdppc\"" Nov 03 22:13:16.394025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.393992 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b"] Nov 03 22:13:16.396251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.396233 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.414441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.414419 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fcbtf"] Nov 03 22:13:16.416392 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.416372 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.418179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.418162 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b"] Nov 03 22:13:16.441252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.440919 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:13:16.441438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.441299 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 22:13:16.453214 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.453185 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0a56fa6-1997-4c34-8b30-fffd023081fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.453346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.453245 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzg8\" (UniqueName: \"kubernetes.io/projected/d0a56fa6-1997-4c34-8b30-fffd023081fb-kube-api-access-rqzg8\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.453346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.453323 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a56fa6-1997-4c34-8b30-fffd023081fb-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.453468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.453363 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0a56fa6-1997-4c34-8b30-fffd023081fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.460261 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.460240 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:13:16.460512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.460483 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 22:13:16.460614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.460517 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:13:16.460614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.460526 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:13:16.460614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.460540 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2cqjh\"" Nov 03 22:13:16.460614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.460548 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xxldp\"" Nov 03 22:13:16.554042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554004 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a56fa6-1997-4c34-8b30-fffd023081fb-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.554042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554044 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-wtmp\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554082 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0a56fa6-1997-4c34-8b30-fffd023081fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.554284 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554268 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q58j\" (UniqueName: \"kubernetes.io/projected/6b8c9424-3148-40cb-ab34-c7cb86921f48-kube-api-access-6q58j\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554294 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-root\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554311 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c9424-3148-40cb-ab34-c7cb86921f48-metrics-client-ca\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554331 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554393 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.554580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554557 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-sys\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554589 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.554686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554619 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0a56fa6-1997-4c34-8b30-fffd023081fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.554833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554687 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.554833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554760 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.554833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554803 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-textfile\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554881 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.554987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554925 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm88l\" (UniqueName: \"kubernetes.io/projected/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-api-access-qm88l\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.554987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554930 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a56fa6-1997-4c34-8b30-fffd023081fb-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.554987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.554970 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-tls\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.555138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.555022 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.555138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.555073 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzg8\" (UniqueName: \"kubernetes.io/projected/d0a56fa6-1997-4c34-8b30-fffd023081fb-kube-api-access-rqzg8\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.556622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.556603 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0a56fa6-1997-4c34-8b30-fffd023081fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.556987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.556968 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0a56fa6-1997-4c34-8b30-fffd023081fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.577426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.577397 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzg8\" (UniqueName: \"kubernetes.io/projected/d0a56fa6-1997-4c34-8b30-fffd023081fb-kube-api-access-rqzg8\") pod \"openshift-state-metrics-7bd7489957-45kb5\" (UID: \"d0a56fa6-1997-4c34-8b30-fffd023081fb\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.650025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.649995 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" Nov 03 22:13:16.656440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656413 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm88l\" (UniqueName: \"kubernetes.io/projected/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-api-access-qm88l\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.656577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656448 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-tls\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656472 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.656577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656499 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-wtmp\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656524 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q58j\" (UniqueName: \"kubernetes.io/projected/6b8c9424-3148-40cb-ab34-c7cb86921f48-kube-api-access-6q58j\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656540 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-root\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656556 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c9424-3148-40cb-ab34-c7cb86921f48-metrics-client-ca\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656575 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656600 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.656928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656621 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-sys\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656641 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.656928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656663 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.656928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656678 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.656928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656695 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-textfile\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.656928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.656736 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.657262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.657004 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-wtmp\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.657329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.657300 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.658209 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.658275 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-sys\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.658476 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-textfile\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.658518 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b8c9424-3148-40cb-ab34-c7cb86921f48-root\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.659076 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.659703 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.660070 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.660251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.660199 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.661112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.661092 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c9424-3148-40cb-ab34-c7cb86921f48-metrics-client-ca\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.661782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.661707 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.661873 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.661862 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b8c9424-3148-40cb-ab34-c7cb86921f48-node-exporter-tls\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.678391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.678361 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm88l\" (UniqueName: \"kubernetes.io/projected/0ccd3e6d-5e97-4a53-b81e-8dde9c536497-kube-api-access-qm88l\") pod \"kube-state-metrics-5dc8d7d89b-mxf8b\" (UID: \"0ccd3e6d-5e97-4a53-b81e-8dde9c536497\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.679112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.679086 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q58j\" (UniqueName: \"kubernetes.io/projected/6b8c9424-3148-40cb-ab34-c7cb86921f48-kube-api-access-6q58j\") pod \"node-exporter-fcbtf\" (UID: \"6b8c9424-3148-40cb-ab34-c7cb86921f48\") " pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.708056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.708024 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" Nov 03 22:13:16.729059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.729035 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcbtf" Nov 03 22:13:16.738131 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:16.738099 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8c9424_3148_40cb_ab34_c7cb86921f48.slice/crio-d99c6fa28257eb0b7ef773a979ad6b076b41e99163221c45f265ab8feafa1496 WatchSource:0}: Error finding container d99c6fa28257eb0b7ef773a979ad6b076b41e99163221c45f265ab8feafa1496: Status 404 returned error can't find the container with id d99c6fa28257eb0b7ef773a979ad6b076b41e99163221c45f265ab8feafa1496 Nov 03 22:13:16.850672 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.850634 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcbtf" event={"ID":"6b8c9424-3148-40cb-ab34-c7cb86921f48","Type":"ContainerStarted","Data":"d99c6fa28257eb0b7ef773a979ad6b076b41e99163221c45f265ab8feafa1496"} Nov 03 22:13:16.967886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:16.967747 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5"] Nov 03 22:13:16.972523 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:16.972493 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a56fa6_1997_4c34_8b30_fffd023081fb.slice/crio-712d4e4d867b28971aba661473acd155041cc38e8dbea535e4af6cc19b9f8482 WatchSource:0}: Error finding container 712d4e4d867b28971aba661473acd155041cc38e8dbea535e4af6cc19b9f8482: Status 404 returned error can't find the container with id 712d4e4d867b28971aba661473acd155041cc38e8dbea535e4af6cc19b9f8482 Nov 03 22:13:17.014962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:17.014934 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b"] Nov 03 22:13:17.018673 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:17.018645 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccd3e6d_5e97_4a53_b81e_8dde9c536497.slice/crio-59b87d5e373d23d523358c8416a538e54f7199de284e5fdca7dd09e3776849e3 WatchSource:0}: Error finding container 59b87d5e373d23d523358c8416a538e54f7199de284e5fdca7dd09e3776849e3: Status 404 returned error can't find the container with id 59b87d5e373d23d523358c8416a538e54f7199de284e5fdca7dd09e3776849e3 Nov 03 22:13:17.856088 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:17.856046 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" event={"ID":"0ccd3e6d-5e97-4a53-b81e-8dde9c536497","Type":"ContainerStarted","Data":"59b87d5e373d23d523358c8416a538e54f7199de284e5fdca7dd09e3776849e3"} Nov 03 22:13:17.858275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:17.858241 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" event={"ID":"d0a56fa6-1997-4c34-8b30-fffd023081fb","Type":"ContainerStarted","Data":"b31929a34e79fa92d2bfc1794c704e72950e2216255692149b0db635ac90d23e"} Nov 03 22:13:17.858399 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:17.858298 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" event={"ID":"d0a56fa6-1997-4c34-8b30-fffd023081fb","Type":"ContainerStarted","Data":"4fe41b08e02acacc230239b5cae79a45ce4d261fb0770ad2a5cb7b57fc075f42"} Nov 03 22:13:17.858399 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:17.858328 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" event={"ID":"d0a56fa6-1997-4c34-8b30-fffd023081fb","Type":"ContainerStarted","Data":"712d4e4d867b28971aba661473acd155041cc38e8dbea535e4af6cc19b9f8482"} Nov 03 22:13:17.860186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:17.860103 4625 generic.go:356] "Generic (PLEG): container finished" podID="6b8c9424-3148-40cb-ab34-c7cb86921f48" containerID="96cafec90e9ee037a1ea9eb6add58d0c219d44d9666a3df65778bc290f7d33ed" exitCode=0 Nov 03 22:13:17.860186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:17.860147 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcbtf" event={"ID":"6b8c9424-3148-40cb-ab34-c7cb86921f48","Type":"ContainerDied","Data":"96cafec90e9ee037a1ea9eb6add58d0c219d44d9666a3df65778bc290f7d33ed"} Nov 03 22:13:18.462565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.462421 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c59b76cb-ksvrz"] Nov 03 22:13:18.465628 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.465606 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.472806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.472510 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49k6b" Nov 03 22:13:18.483127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.482937 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 22:13:18.483127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.482991 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 22:13:18.483127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.483061 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-skb8h\"" Nov 03 22:13:18.483127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.483071 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 22:13:18.483127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.483098 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8um654qtnarr6\"" Nov 03 22:13:18.483444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.483145 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 22:13:18.483444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.483178 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 22:13:18.495563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.495534 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c59b76cb-ksvrz"] Nov 03 22:13:18.569556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569521 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-grpc-tls\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.569556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569557 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d2e3700-a62d-4a9b-a836-8771e39b0014-metrics-client-ca\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.569816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569582 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.569816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569639 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.569816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569766 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d2e3700-a62d-4a9b-a836-8771e39b0014-tmp\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.569931 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569904 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.569981 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569965 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.570035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.569997 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2z5w\" (UniqueName: \"kubernetes.io/projected/5d2e3700-a62d-4a9b-a836-8771e39b0014-kube-api-access-g2z5w\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.570128 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.570109 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-tls\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671253 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671300 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671327 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2z5w\" (UniqueName: \"kubernetes.io/projected/5d2e3700-a62d-4a9b-a836-8771e39b0014-kube-api-access-g2z5w\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671359 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-tls\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671383 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-grpc-tls\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671409 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d2e3700-a62d-4a9b-a836-8771e39b0014-metrics-client-ca\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671441 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671466 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.671772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671516 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d2e3700-a62d-4a9b-a836-8771e39b0014-tmp\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.672020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.671944 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d2e3700-a62d-4a9b-a836-8771e39b0014-tmp\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.672247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.672199 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d2e3700-a62d-4a9b-a836-8771e39b0014-metrics-client-ca\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.674178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.674130 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.674363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.674339 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.674633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.674610 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-tls\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.674670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.674640 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-grpc-tls\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.674974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.674956 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.675093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.675072 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d2e3700-a62d-4a9b-a836-8771e39b0014-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.693765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.693735 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2z5w\" (UniqueName: \"kubernetes.io/projected/5d2e3700-a62d-4a9b-a836-8771e39b0014-kube-api-access-g2z5w\") pod \"thanos-querier-6c59b76cb-ksvrz\" (UID: \"5d2e3700-a62d-4a9b-a836-8771e39b0014\") " pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.784949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.784871 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:18.865466 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.865422 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" event={"ID":"d0a56fa6-1997-4c34-8b30-fffd023081fb","Type":"ContainerStarted","Data":"03d7cd64ae45abe32e1b24d0e885422d6c56a4c9088bf8e7d002ab4aa19a028b"} Nov 03 22:13:18.867865 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.867834 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcbtf" event={"ID":"6b8c9424-3148-40cb-ab34-c7cb86921f48","Type":"ContainerStarted","Data":"31d6af67c98785adc4d109b01e04474fa8576afffa4219a827a0f6bb5c1298a6"} Nov 03 22:13:18.867986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.867884 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcbtf" event={"ID":"6b8c9424-3148-40cb-ab34-c7cb86921f48","Type":"ContainerStarted","Data":"86870dd15a8eb618172a222488e90f096de4c39a7c043f757dc17b9dcc9df701"} Nov 03 22:13:18.870163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.870141 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" event={"ID":"0ccd3e6d-5e97-4a53-b81e-8dde9c536497","Type":"ContainerStarted","Data":"3b4a8fcad72027145cc8575ff787987621481f652af668a1d55079aa08146a01"} Nov 03 22:13:18.870266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.870188 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" event={"ID":"0ccd3e6d-5e97-4a53-b81e-8dde9c536497","Type":"ContainerStarted","Data":"20d16f9bfe3a8c658be675f85f7bbc512f08539dfcea965502188207749edd91"} Nov 03 22:13:18.870266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.870216 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" event={"ID":"0ccd3e6d-5e97-4a53-b81e-8dde9c536497","Type":"ContainerStarted","Data":"cda497adb6d06edcb17a21f20a90613f179b027a3d1aa575bf50751673ce83a7"} Nov 03 22:13:18.906479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.906435 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-45kb5" podStartSLOduration=2.04714288 podStartE2EDuration="2.906424036s" totalImagesPullingTime="859.281156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:18.905840713 +0000 UTC m=+20.335176703" watchObservedRunningTime="2026-11-03 22:13:18.906424036 +0000 UTC m=+20.335760028" Nov 03 22:13:18.946855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.946806 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mxf8b" podStartSLOduration=1.77729502 podStartE2EDuration="2.946793268s" totalImagesPullingTime="1.169498248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:18.946546408 +0000 UTC m=+20.375882401" watchObservedRunningTime="2026-11-03 22:13:18.946793268 +0000 UTC m=+20.376129254" Nov 03 22:13:18.992010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:18.991956 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fcbtf" podStartSLOduration=2.0391944 podStartE2EDuration="2.991941287s" totalImagesPullingTime="871.191092ms" totalInitContainerRuntime="81.555795ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:18.990571161 +0000 UTC m=+20.419907179" watchObservedRunningTime="2026-11-03 22:13:18.991941287 +0000 UTC m=+20.421277280" Nov 03 22:13:19.105061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:19.105022 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c59b76cb-ksvrz"] Nov 03 22:13:19.109037 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:19.109012 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2e3700_a62d_4a9b_a836_8771e39b0014.slice/crio-18e1c23a73304d310a0c7c2b20655fe7a5a28b1114d5e1aa78ac00e1eaba5ffb WatchSource:0}: Error finding container 18e1c23a73304d310a0c7c2b20655fe7a5a28b1114d5e1aa78ac00e1eaba5ffb: Status 404 returned error can't find the container with id 18e1c23a73304d310a0c7c2b20655fe7a5a28b1114d5e1aa78ac00e1eaba5ffb Nov 03 22:13:19.828447 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:19.828415 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4hrjm" Nov 03 22:13:19.855168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:19.855135 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:19.855316 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:19.855199 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:19.860586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:19.860563 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:19.876778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:19.875688 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" event={"ID":"5d2e3700-a62d-4a9b-a836-8771e39b0014","Type":"ContainerStarted","Data":"18e1c23a73304d310a0c7c2b20655fe7a5a28b1114d5e1aa78ac00e1eaba5ffb"} Nov 03 22:13:19.882251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:19.882228 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:13:20.849515 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.849487 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-dd945bbc8-ggpxf"] Nov 03 22:13:20.852251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.852234 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:20.866798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.866773 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:13:20.878022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.877945 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 22:13:20.878022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.877970 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 22:13:20.878022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.877985 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g7w8t\"" Nov 03 22:13:20.878022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.877992 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-26jl7oroc6teg\"" Nov 03 22:13:20.878599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.878037 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 22:13:20.879411 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.879387 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dd945bbc8-ggpxf"] Nov 03 22:13:20.991143 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.991117 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-audit-log\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:20.991292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.991170 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-secret-metrics-server-client-certs\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:20.991292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.991192 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:20.991412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.991359 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqkr\" (UniqueName: \"kubernetes.io/projected/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-kube-api-access-ngqkr\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:20.991412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.991409 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-secret-metrics-server-tls\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:20.991525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.991474 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-client-ca-bundle\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:20.991703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:20.991679 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-metrics-server-audit-profiles\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.093195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093113 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-metrics-server-audit-profiles\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.093195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093167 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-audit-log\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.093370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093215 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-secret-metrics-server-client-certs\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.093370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093254 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.093370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093304 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqkr\" (UniqueName: \"kubernetes.io/projected/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-kube-api-access-ngqkr\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.093370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093358 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-secret-metrics-server-tls\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.093754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093684 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-client-ca-bundle\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.094001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.093955 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-audit-log\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.094092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.094016 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.094183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.094162 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-metrics-server-audit-profiles\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.096143 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.096123 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-client-ca-bundle\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.096367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.096349 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-secret-metrics-server-tls\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.096689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.096674 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-secret-metrics-server-client-certs\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.116227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.116199 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn"] Nov 03 22:13:21.116459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.116436 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqkr\" (UniqueName: \"kubernetes.io/projected/9998718c-eddb-4aef-ac0d-2f15aa6b48ea-kube-api-access-ngqkr\") pod \"metrics-server-dd945bbc8-ggpxf\" (UID: \"9998718c-eddb-4aef-ac0d-2f15aa6b48ea\") " pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.120185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.120013 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.137022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.136993 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 22:13:21.137022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.137020 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xpjnp\"" Nov 03 22:13:21.138168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.138143 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn"] Nov 03 22:13:21.227851 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.227814 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:21.295529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.295387 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3b4bc26-994e-4ad1-b961-83e96121136a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nb6fn\" (UID: \"c3b4bc26-994e-4ad1-b961-83e96121136a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.295756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.295667 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csss8\" (UniqueName: \"kubernetes.io/projected/c3b4bc26-994e-4ad1-b961-83e96121136a-kube-api-access-csss8\") pod \"monitoring-plugin-77f4f76594-nb6fn\" (UID: \"c3b4bc26-994e-4ad1-b961-83e96121136a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.400137 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.396795 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csss8\" (UniqueName: \"kubernetes.io/projected/c3b4bc26-994e-4ad1-b961-83e96121136a-kube-api-access-csss8\") pod \"monitoring-plugin-77f4f76594-nb6fn\" (UID: \"c3b4bc26-994e-4ad1-b961-83e96121136a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.400137 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.396878 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3b4bc26-994e-4ad1-b961-83e96121136a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nb6fn\" (UID: \"c3b4bc26-994e-4ad1-b961-83e96121136a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.401330 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.401298 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3b4bc26-994e-4ad1-b961-83e96121136a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nb6fn\" (UID: \"c3b4bc26-994e-4ad1-b961-83e96121136a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.421102 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.421074 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csss8\" (UniqueName: \"kubernetes.io/projected/c3b4bc26-994e-4ad1-b961-83e96121136a-kube-api-access-csss8\") pod \"monitoring-plugin-77f4f76594-nb6fn\" (UID: \"c3b4bc26-994e-4ad1-b961-83e96121136a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.433850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.433825 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:21.544493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.544453 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dd945bbc8-ggpxf"] Nov 03 22:13:21.548888 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:21.548857 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9998718c_eddb_4aef_ac0d_2f15aa6b48ea.slice/crio-d669fca3289f1d02213e22a1169a6e86bcd7b48fbc1c0bb516b19bead1c80417 WatchSource:0}: Error finding container d669fca3289f1d02213e22a1169a6e86bcd7b48fbc1c0bb516b19bead1c80417: Status 404 returned error can't find the container with id d669fca3289f1d02213e22a1169a6e86bcd7b48fbc1c0bb516b19bead1c80417 Nov 03 22:13:21.738489 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.738462 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn"] Nov 03 22:13:21.742677 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:21.742640 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b4bc26_994e_4ad1_b961_83e96121136a.slice/crio-ee1da4640ed7ed768401adc41aa2b66395f9aa1e862013ba79c5303ab9e05703 WatchSource:0}: Error finding container ee1da4640ed7ed768401adc41aa2b66395f9aa1e862013ba79c5303ab9e05703: Status 404 returned error can't find the container with id ee1da4640ed7ed768401adc41aa2b66395f9aa1e862013ba79c5303ab9e05703 Nov 03 22:13:21.883785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.883749 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" event={"ID":"c3b4bc26-994e-4ad1-b961-83e96121136a","Type":"ContainerStarted","Data":"ee1da4640ed7ed768401adc41aa2b66395f9aa1e862013ba79c5303ab9e05703"} Nov 03 22:13:21.886440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.886412 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" event={"ID":"5d2e3700-a62d-4a9b-a836-8771e39b0014","Type":"ContainerStarted","Data":"853959b43bf72631be1104eb18baa2373e5c95d9d0dc49c5069a8c30b3b74025"} Nov 03 22:13:21.886547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.886471 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" event={"ID":"5d2e3700-a62d-4a9b-a836-8771e39b0014","Type":"ContainerStarted","Data":"1a4828e32b9c9799a0f047ba6dbd2727717eb484a630e345e5a07d2a181e7a98"} Nov 03 22:13:21.886547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.886512 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" event={"ID":"5d2e3700-a62d-4a9b-a836-8771e39b0014","Type":"ContainerStarted","Data":"e1b604d345bd5e057d3c7684c46c5e3243446836d4c33691309dbb7e9165f72d"} Nov 03 22:13:21.886627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.886548 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" event={"ID":"5d2e3700-a62d-4a9b-a836-8771e39b0014","Type":"ContainerStarted","Data":"b85cb13d596deb7824358781e2bba367a1d23fa3739fc5565b2436637311f28f"} Nov 03 22:13:21.886627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.886577 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" event={"ID":"5d2e3700-a62d-4a9b-a836-8771e39b0014","Type":"ContainerStarted","Data":"78fb68c44519d910241709031242cf7c6741eed66020a1add0895be36833e09d"} Nov 03 22:13:21.887492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:21.887473 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" event={"ID":"9998718c-eddb-4aef-ac0d-2f15aa6b48ea","Type":"ContainerStarted","Data":"d669fca3289f1d02213e22a1169a6e86bcd7b48fbc1c0bb516b19bead1c80417"} Nov 03 22:13:22.895099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:22.895062 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" event={"ID":"5d2e3700-a62d-4a9b-a836-8771e39b0014","Type":"ContainerStarted","Data":"b4ff3e870facd2ae9d49673cd1f27306ed8863cd3db96ba913f94f978b203902"} Nov 03 22:13:22.895505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:22.895269 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:22.963997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:22.963948 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" podStartSLOduration=2.5664110730000003 podStartE2EDuration="4.963933615s" totalImagesPullingTime="2.397522542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:22.96377748 +0000 UTC m=+24.393113473" watchObservedRunningTime="2026-11-03 22:13:22.963933615 +0000 UTC m=+24.393269604" Nov 03 22:13:23.900589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:23.900551 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" event={"ID":"c3b4bc26-994e-4ad1-b961-83e96121136a","Type":"ContainerStarted","Data":"7b877d19285b4c6bde533168d912f611612421556616c217708d13809d8f34dd"} Nov 03 22:13:23.901161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:23.900617 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:23.902319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:23.902282 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" event={"ID":"9998718c-eddb-4aef-ac0d-2f15aa6b48ea","Type":"ContainerStarted","Data":"7030f3202be5b7cab0a404d9519804c4fe0da2d52617a3ed6edb1166f1edd09a"} Nov 03 22:13:23.905761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:23.905738 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" Nov 03 22:13:23.940563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:23.940515 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nb6fn" podStartSLOduration=1.375210487 podStartE2EDuration="2.940500096s" totalImagesPullingTime="1.565289609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:23.93975693 +0000 UTC m=+25.369092924" watchObservedRunningTime="2026-11-03 22:13:23.940500096 +0000 UTC m=+25.369836089" Nov 03 22:13:24.019799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.019678 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" podStartSLOduration=2.26385735 podStartE2EDuration="4.019662687s" totalImagesPullingTime="1.755805337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:24.019424157 +0000 UTC m=+25.448760174" watchObservedRunningTime="2026-11-03 22:13:24.019662687 +0000 UTC m=+25.448998677" Nov 03 22:13:24.615874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.615838 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d49ff4b4c-8f5hj"] Nov 03 22:13:24.645318 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.645288 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-596475dbc7-mfwdg"] Nov 03 22:13:24.673185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.673157 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596475dbc7-mfwdg"] Nov 03 22:13:24.673331 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.673270 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.717512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.717480 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:24.734286 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734260 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-service-ca\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.734439 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734298 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-oauth-config\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.734439 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734315 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-oauth-serving-cert\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.734439 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734339 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-trusted-ca-bundle\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.734550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734440 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxmc\" (UniqueName: \"kubernetes.io/projected/faa4cd56-6faf-4b8e-a6c3-39eafe415014-kube-api-access-snxmc\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.734550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734504 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa4cd56-6faf-4b8e-a6c3-39eafe415014-tmp\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.734550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734547 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-serving-cert\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.734662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.734562 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-config\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835398 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa4cd56-6faf-4b8e-a6c3-39eafe415014-tmp\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835444 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-serving-cert\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835461 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-config\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835482 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-service-ca\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835507 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-oauth-config\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835521 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-oauth-serving-cert\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835550 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-trusted-ca-bundle\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.835593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.835572 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxmc\" (UniqueName: \"kubernetes.io/projected/faa4cd56-6faf-4b8e-a6c3-39eafe415014-kube-api-access-snxmc\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.836043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.836019 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa4cd56-6faf-4b8e-a6c3-39eafe415014-tmp\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.836353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.836323 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-config\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.836443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.836412 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-service-ca\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.836483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.836450 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-oauth-serving-cert\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.836517 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.836494 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-trusted-ca-bundle\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.838503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.838479 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-oauth-config\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.838585 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.838529 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-serving-cert\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.858193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.858171 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxmc\" (UniqueName: \"kubernetes.io/projected/faa4cd56-6faf-4b8e-a6c3-39eafe415014-kube-api-access-snxmc\") pod \"console-596475dbc7-mfwdg\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:24.994259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:24.994183 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:25.352095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:25.352057 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596475dbc7-mfwdg"] Nov 03 22:13:25.356153 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:25.356127 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa4cd56_6faf_4b8e_a6c3_39eafe415014.slice/crio-b916a095a2f91948316c9242bdc8dbd21fe7b75488cc07cd3e1c29b601b0ad3e WatchSource:0}: Error finding container b916a095a2f91948316c9242bdc8dbd21fe7b75488cc07cd3e1c29b601b0ad3e: Status 404 returned error can't find the container with id b916a095a2f91948316c9242bdc8dbd21fe7b75488cc07cd3e1c29b601b0ad3e Nov 03 22:13:25.911502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:25.911466 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596475dbc7-mfwdg" event={"ID":"faa4cd56-6faf-4b8e-a6c3-39eafe415014","Type":"ContainerStarted","Data":"103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a"} Nov 03 22:13:25.911699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:25.911518 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596475dbc7-mfwdg" event={"ID":"faa4cd56-6faf-4b8e-a6c3-39eafe415014","Type":"ContainerStarted","Data":"b916a095a2f91948316c9242bdc8dbd21fe7b75488cc07cd3e1c29b601b0ad3e"} Nov 03 22:13:25.955241 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:25.955191 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-596475dbc7-mfwdg" podStartSLOduration=1.9551770469999998 podStartE2EDuration="1.955177047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:25.954264981 +0000 UTC m=+27.383601000" watchObservedRunningTime="2026-11-03 22:13:25.955177047 +0000 UTC m=+27.384513040" Nov 03 22:13:27.482547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:27.482518 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xpshj" Nov 03 22:13:28.909621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:28.909587 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c59b76cb-ksvrz" Nov 03 22:13:29.029231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.029196 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-596475dbc7-mfwdg"] Nov 03 22:13:29.073481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.073445 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5895c66dc9-bh25s"] Nov 03 22:13:29.076120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.076104 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.102698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.102673 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5895c66dc9-bh25s"] Nov 03 22:13:29.174026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.173948 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-oauth-serving-cert\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.174026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.173987 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-config\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.174026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.174008 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-oauth-config\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.174228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.174152 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-serving-cert\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.174228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.174190 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-trusted-ca-bundle\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.174292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.174246 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-service-ca\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.174325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.174306 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-tmp\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.174359 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.174324 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slb4x\" (UniqueName: \"kubernetes.io/projected/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-kube-api-access-slb4x\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275746 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-oauth-serving-cert\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275807 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-config\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275827 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-oauth-config\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275851 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-serving-cert\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275870 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-trusted-ca-bundle\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275896 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-service-ca\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275920 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-tmp\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.275961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.275948 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slb4x\" (UniqueName: \"kubernetes.io/projected/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-kube-api-access-slb4x\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.276429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.276375 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-tmp\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.276652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.276629 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-oauth-serving-cert\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.276752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.276688 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-service-ca\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.276963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.276939 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-config\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.277040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.277022 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-trusted-ca-bundle\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.278578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.278550 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-oauth-config\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.278663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.278648 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-serving-cert\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.298451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.298412 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slb4x\" (UniqueName: \"kubernetes.io/projected/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-kube-api-access-slb4x\") pod \"console-5895c66dc9-bh25s\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.387882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.387842 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:29.695458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.695373 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5895c66dc9-bh25s"] Nov 03 22:13:29.699399 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:13:29.699372 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a88332_8fdd_4c0d_8f3e_4578d6932bb5.slice/crio-bf0b991b1a909b3fe98421ee23c3b8842c1ba49dd4184cd3f5e3b09640633ffb WatchSource:0}: Error finding container bf0b991b1a909b3fe98421ee23c3b8842c1ba49dd4184cd3f5e3b09640633ffb: Status 404 returned error can't find the container with id bf0b991b1a909b3fe98421ee23c3b8842c1ba49dd4184cd3f5e3b09640633ffb Nov 03 22:13:29.927695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.927660 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895c66dc9-bh25s" event={"ID":"86a88332-8fdd-4c0d-8f3e-4578d6932bb5","Type":"ContainerStarted","Data":"74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10"} Nov 03 22:13:29.928140 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.927709 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895c66dc9-bh25s" event={"ID":"86a88332-8fdd-4c0d-8f3e-4578d6932bb5","Type":"ContainerStarted","Data":"bf0b991b1a909b3fe98421ee23c3b8842c1ba49dd4184cd3f5e3b09640633ffb"} Nov 03 22:13:29.969592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:29.969473 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5895c66dc9-bh25s" podStartSLOduration=0.969458024 podStartE2EDuration="969.458024ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:29.968256844 +0000 UTC m=+31.397592836" watchObservedRunningTime="2026-11-03 22:13:29.969458024 +0000 UTC m=+31.398794053" Nov 03 22:13:32.192145 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:32.192106 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:13:34.995117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:34.995074 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:39.388227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:39.388193 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:39.388735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:39.388280 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:39.392904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:39.392879 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:39.968589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:39.968558 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:13:40.043635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:40.043595 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f7df997d-hgx6n"] Nov 03 22:13:41.228662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:41.228567 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:41.228662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:41.228637 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:13:41.853150 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:41.853114 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q8k65-h5z45"] Nov 03 22:13:41.853352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:41.853326 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-q8k65-h5z45" podUID="acca86e6-473d-442d-9ca9-8ba949d8caa9" containerName="insights-gathering" containerID="cri-o://4f95cd10477843871f654653bfdaa2b63967e4cb93a32e115a476078d9c9310a" gracePeriod=30 Nov 03 22:13:41.973684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:41.973651 4625 generic.go:356] "Generic (PLEG): container finished" podID="acca86e6-473d-442d-9ca9-8ba949d8caa9" containerID="4f95cd10477843871f654653bfdaa2b63967e4cb93a32e115a476078d9c9310a" exitCode=2 Nov 03 22:13:41.973812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:41.973730 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8k65-h5z45" event={"ID":"acca86e6-473d-442d-9ca9-8ba949d8caa9","Type":"ContainerDied","Data":"4f95cd10477843871f654653bfdaa2b63967e4cb93a32e115a476078d9c9310a"} Nov 03 22:13:42.093427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.093404 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:13:42.202650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.202569 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\") pod \"acca86e6-473d-442d-9ca9-8ba949d8caa9\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " Nov 03 22:13:42.202829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.202662 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\") pod \"acca86e6-473d-442d-9ca9-8ba949d8caa9\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " Nov 03 22:13:42.202829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.202737 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\" (UniqueName: \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\") pod \"acca86e6-473d-442d-9ca9-8ba949d8caa9\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " Nov 03 22:13:42.202923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.202847 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\") pod \"acca86e6-473d-442d-9ca9-8ba949d8caa9\" (UID: \"acca86e6-473d-442d-9ca9-8ba949d8caa9\") " Nov 03 22:13:42.203016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.202991 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path" pod "acca86e6-473d-442d-9ca9-8ba949d8caa9" (UID: "acca86e6-473d-442d-9ca9-8ba949d8caa9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:13:42.203182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.203157 4625 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/acca86e6-473d-442d-9ca9-8ba949d8caa9-archives-path\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:42.203182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.203161 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle" pod "acca86e6-473d-442d-9ca9-8ba949d8caa9" (UID: "acca86e6-473d-442d-9ca9-8ba949d8caa9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:42.203316 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.203272 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle" pod "acca86e6-473d-442d-9ca9-8ba949d8caa9" (UID: "acca86e6-473d-442d-9ca9-8ba949d8caa9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:42.205097 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.205077 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8" pod "acca86e6-473d-442d-9ca9-8ba949d8caa9" (UID: "acca86e6-473d-442d-9ca9-8ba949d8caa9"). InnerVolumeSpecName "kube-api-access-xgwh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:42.304389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.304350 4625 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-service-ca-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:42.304389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.304387 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgwh8\" (UniqueName: \"kubernetes.io/projected/acca86e6-473d-442d-9ca9-8ba949d8caa9-kube-api-access-xgwh8\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:42.304823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.304399 4625 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acca86e6-473d-442d-9ca9-8ba949d8caa9-trusted-ca-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:42.979312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.979283 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8k65-h5z45" Nov 03 22:13:42.979479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.979284 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8k65-h5z45" event={"ID":"acca86e6-473d-442d-9ca9-8ba949d8caa9","Type":"ContainerDied","Data":"34266ea37bcf4166f6ce89374b304431bbc9c9e35f456cabb7f790300d489cf3"} Nov 03 22:13:42.979479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:42.979432 4625 scope.go:122] "RemoveContainer" containerID="4f95cd10477843871f654653bfdaa2b63967e4cb93a32e115a476078d9c9310a" Nov 03 22:13:43.026673 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:43.026637 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q8k65-h5z45"] Nov 03 22:13:43.037595 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:43.037566 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-q8k65-h5z45"] Nov 03 22:13:44.724747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:44.724699 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acca86e6-473d-442d-9ca9-8ba949d8caa9" path="/var/lib/kubelet/pods/acca86e6-473d-442d-9ca9-8ba949d8caa9/volumes" Nov 03 22:13:49.635441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.635402 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d49ff4b4c-8f5hj" podUID="428eb929-6f4b-41b9-8452-606272ed7650" containerName="console" containerID="cri-o://5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951" gracePeriod=15 Nov 03 22:13:49.892043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.891989 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:49.968074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968042 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-console-config\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-console-config\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968090 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-service-ca\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-service-ca\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968108 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-serving-cert\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968224 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428eb929-6f4b-41b9-8452-606272ed7650-kube-api-access-4x6jx\" (UniqueName: \"kubernetes.io/projected/428eb929-6f4b-41b9-8452-606272ed7650-kube-api-access-4x6jx\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968319 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-oauth-config\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968371 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-oauth-serving-cert\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968434 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428eb929-6f4b-41b9-8452-606272ed7650-tmp\" (UniqueName: \"kubernetes.io/empty-dir/428eb929-6f4b-41b9-8452-606272ed7650-tmp\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968452 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-console-config" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:49.968529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968474 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-trusted-ca-bundle\") pod \"428eb929-6f4b-41b9-8452-606272ed7650\" (UID: \"428eb929-6f4b-41b9-8452-606272ed7650\") " Nov 03 22:13:49.968529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968518 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-service-ca" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:49.968785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968766 4625 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-console-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:49.968847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968791 4625 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:49.968847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968826 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428eb929-6f4b-41b9-8452-606272ed7650-tmp" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:13:49.968955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.968866 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-oauth-serving-cert" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:49.969126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.969107 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-trusted-ca-bundle" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:49.970509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.970482 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-serving-cert" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:49.970633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.970504 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-oauth-config" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:49.970633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:49.970571 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428eb929-6f4b-41b9-8452-606272ed7650-kube-api-access-4x6jx" pod "428eb929-6f4b-41b9-8452-606272ed7650" (UID: "428eb929-6f4b-41b9-8452-606272ed7650"). InnerVolumeSpecName "kube-api-access-4x6jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:50.010145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.010110 4625 generic.go:356] "Generic (PLEG): container finished" podID="428eb929-6f4b-41b9-8452-606272ed7650" containerID="5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951" exitCode=0 Nov 03 22:13:50.010308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.010153 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d49ff4b4c-8f5hj" event={"ID":"428eb929-6f4b-41b9-8452-606272ed7650","Type":"ContainerDied","Data":"5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951"} Nov 03 22:13:50.010308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.010168 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d49ff4b4c-8f5hj" Nov 03 22:13:50.010308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.010191 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d49ff4b4c-8f5hj" event={"ID":"428eb929-6f4b-41b9-8452-606272ed7650","Type":"ContainerDied","Data":"c2c4c12882f3efcca9ba0bd717e6184010e2b5f7301a49f8a3e97bd617d4ac6c"} Nov 03 22:13:50.010308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.010211 4625 scope.go:122] "RemoveContainer" containerID="5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951" Nov 03 22:13:50.018485 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.018469 4625 scope.go:122] "RemoveContainer" containerID="5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951" Nov 03 22:13:50.018786 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:50.018764 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951\": container with ID starting with 5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951 not found: ID does not exist" containerID="5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951" Nov 03 22:13:50.018840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.018794 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951"} err="failed to get container status \"5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951\": rpc error: code = NotFound desc = could not find container \"5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951\": container with ID starting with 5f52dc6fabf097a2337b5d5fe77a9421b68f56bb3b46170e0781c38ddc8fd951 not found: ID does not exist" Nov 03 22:13:50.056546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.056513 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d49ff4b4c-8f5hj"] Nov 03 22:13:50.060599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.060564 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d49ff4b4c-8f5hj"] Nov 03 22:13:50.069664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.069641 4625 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428eb929-6f4b-41b9-8452-606272ed7650-tmp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:50.069664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.069662 4625 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-trusted-ca-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:50.069835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.069673 4625 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:50.069835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.069684 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x6jx\" (UniqueName: \"kubernetes.io/projected/428eb929-6f4b-41b9-8452-606272ed7650-kube-api-access-4x6jx\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:50.069835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.069693 4625 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/428eb929-6f4b-41b9-8452-606272ed7650-console-oauth-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:50.069835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.069701 4625 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/428eb929-6f4b-41b9-8452-606272ed7650-oauth-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:50.724637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:50.724599 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428eb929-6f4b-41b9-8452-606272ed7650" path="/var/lib/kubelet/pods/428eb929-6f4b-41b9-8452-606272ed7650/volumes" Nov 03 22:13:54.049286 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.049238 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-596475dbc7-mfwdg" podUID="faa4cd56-6faf-4b8e-a6c3-39eafe415014" containerName="console" containerID="cri-o://103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a" gracePeriod=15 Nov 03 22:13:54.332126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.332091 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:54.511599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511555 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-oauth-config\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.511599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511600 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-serving-cert\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.511894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511632 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-config\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-config\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.511894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511762 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-oauth-serving-cert\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.511894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511855 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-trusted-ca-bundle\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.511894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511885 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-service-ca\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.512113 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511903 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa4cd56-6faf-4b8e-a6c3-39eafe415014-kube-api-access-snxmc\" (UniqueName: \"kubernetes.io/projected/faa4cd56-6faf-4b8e-a6c3-39eafe415014-kube-api-access-snxmc\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.512113 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.511931 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa4cd56-6faf-4b8e-a6c3-39eafe415014-tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa4cd56-6faf-4b8e-a6c3-39eafe415014-tmp\") pod \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\" (UID: \"faa4cd56-6faf-4b8e-a6c3-39eafe415014\") " Nov 03 22:13:54.512230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.512162 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-config" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:54.512230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.512170 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-oauth-serving-cert" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:54.512424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.512369 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa4cd56-6faf-4b8e-a6c3-39eafe415014-tmp" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:13:54.512424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.512376 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-service-ca" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:54.512424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.512294 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-trusted-ca-bundle" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:54.513839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.513807 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-oauth-config" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:54.513947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.513902 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-serving-cert" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:54.514069 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.514040 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa4cd56-6faf-4b8e-a6c3-39eafe415014-kube-api-access-snxmc" pod "faa4cd56-6faf-4b8e-a6c3-39eafe415014" (UID: "faa4cd56-6faf-4b8e-a6c3-39eafe415014"). InnerVolumeSpecName "kube-api-access-snxmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:54.613456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613419 4625 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-trusted-ca-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:54.613456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613449 4625 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:54.613456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613464 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snxmc\" (UniqueName: \"kubernetes.io/projected/faa4cd56-6faf-4b8e-a6c3-39eafe415014-kube-api-access-snxmc\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:54.613700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613477 4625 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa4cd56-6faf-4b8e-a6c3-39eafe415014-tmp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:54.613700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613489 4625 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-oauth-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:54.613700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613502 4625 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:54.613700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613512 4625 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-console-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:54.613700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:54.613523 4625 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa4cd56-6faf-4b8e-a6c3-39eafe415014-oauth-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:13:55.029844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.029761 4625 generic.go:356] "Generic (PLEG): container finished" podID="faa4cd56-6faf-4b8e-a6c3-39eafe415014" containerID="103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a" exitCode=0 Nov 03 22:13:55.029844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.029822 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596475dbc7-mfwdg" Nov 03 22:13:55.030018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.029848 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596475dbc7-mfwdg" event={"ID":"faa4cd56-6faf-4b8e-a6c3-39eafe415014","Type":"ContainerDied","Data":"103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a"} Nov 03 22:13:55.030018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.029906 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596475dbc7-mfwdg" event={"ID":"faa4cd56-6faf-4b8e-a6c3-39eafe415014","Type":"ContainerDied","Data":"b916a095a2f91948316c9242bdc8dbd21fe7b75488cc07cd3e1c29b601b0ad3e"} Nov 03 22:13:55.030018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.029931 4625 scope.go:122] "RemoveContainer" containerID="103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a" Nov 03 22:13:55.037891 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.037873 4625 scope.go:122] "RemoveContainer" containerID="103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a" Nov 03 22:13:55.038151 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:13:55.038133 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a\": container with ID starting with 103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a not found: ID does not exist" containerID="103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a" Nov 03 22:13:55.038207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.038159 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a"} err="failed to get container status \"103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a\": rpc error: code = NotFound desc = could not find container \"103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a\": container with ID starting with 103f063c3776ccea0b02e8e2c12cb010e8bf6d07e68e6ea9426ce38634fa7a7a not found: ID does not exist" Nov 03 22:13:55.077648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.077617 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-596475dbc7-mfwdg"] Nov 03 22:13:55.088918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:55.088895 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-596475dbc7-mfwdg"] Nov 03 22:13:56.725363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:13:56.725329 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa4cd56-6faf-4b8e-a6c3-39eafe415014" path="/var/lib/kubelet/pods/faa4cd56-6faf-4b8e-a6c3-39eafe415014/volumes" Nov 03 22:14:01.233116 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:01.233089 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:14:01.237133 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:01.237111 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-dd945bbc8-ggpxf" Nov 03 22:14:02.184906 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:14:02.184864 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:14:05.063295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.063230 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79f7df997d-hgx6n" podUID="b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" containerName="console" containerID="cri-o://edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe" gracePeriod=15 Nov 03 22:14:05.306694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.306663 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cb57fd44f-zqgjf"] Nov 03 22:14:05.307001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.306986 4625 cpu_manager.go:414] "containerMap: removing container" podUID="acca86e6-473d-442d-9ca9-8ba949d8caa9" containerName="insights-gathering" Nov 03 22:14:05.307059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307002 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acca86e6-473d-442d-9ca9-8ba949d8caa9" containerName="insights-gathering" Nov 03 22:14:05.307059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307019 4625 cpu_manager.go:414] "containerMap: removing container" podUID="428eb929-6f4b-41b9-8452-606272ed7650" containerName="console" Nov 03 22:14:05.307059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307024 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428eb929-6f4b-41b9-8452-606272ed7650" containerName="console" Nov 03 22:14:05.307059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307040 4625 cpu_manager.go:414] "containerMap: removing container" podUID="faa4cd56-6faf-4b8e-a6c3-39eafe415014" containerName="console" Nov 03 22:14:05.307059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307045 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa4cd56-6faf-4b8e-a6c3-39eafe415014" containerName="console" Nov 03 22:14:05.307206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307092 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="acca86e6-473d-442d-9ca9-8ba949d8caa9" containerName="insights-gathering" Nov 03 22:14:05.307206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307101 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="428eb929-6f4b-41b9-8452-606272ed7650" containerName="console" Nov 03 22:14:05.307206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307109 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa4cd56-6faf-4b8e-a6c3-39eafe415014" containerName="console" Nov 03 22:14:05.307593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.307572 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:14:05.308593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.308576 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.334209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.334176 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cb57fd44f-zqgjf"] Nov 03 22:14:05.498491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498446 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-config\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-config\") pod \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " Nov 03 22:14:05.498491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498502 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-kube-api-access-hw2xr\" (UniqueName: \"kubernetes.io/projected/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-kube-api-access-hw2xr\") pod \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " Nov 03 22:14:05.498779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498525 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-serving-cert\") pod \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " Nov 03 22:14:05.498779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498561 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-oauth-serving-cert\") pod \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " Nov 03 22:14:05.498779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498592 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-oauth-config\") pod \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " Nov 03 22:14:05.498779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498634 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-tmp\") pod \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " Nov 03 22:14:05.498779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498672 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-service-ca\") pod \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\" (UID: \"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32\") " Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498840 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-oauth-serving-cert\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498872 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-console-config\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498920 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95825889-a81f-4738-a105-1ff28b59e21f-console-serving-cert\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498960 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95825889-a81f-4738-a105-1ff28b59e21f-console-oauth-config\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498981 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-tmp" pod "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" (UID: "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498994 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95825889-a81f-4738-a105-1ff28b59e21f-tmp\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498989 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-config" pod "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" (UID: "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:05.499046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.498995 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-oauth-serving-cert" pod "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" (UID: "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:05.499354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499078 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-trusted-ca-bundle\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499323 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-service-ca" pod "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" (UID: "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:05.499428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499361 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-service-ca\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499408 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6tw\" (UniqueName: \"kubernetes.io/projected/95825889-a81f-4738-a105-1ff28b59e21f-kube-api-access-wf6tw\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.499521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499502 4625 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-oauth-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:05.499558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499524 4625 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-tmp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:05.499558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499541 4625 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:05.499558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.499555 4625 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:05.500875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.500846 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-oauth-config" pod "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" (UID: "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:05.500984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.500911 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-serving-cert" pod "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" (UID: "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:05.500984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.500918 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-kube-api-access-hw2xr" pod "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" (UID: "b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32"). InnerVolumeSpecName "kube-api-access-hw2xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:05.600118 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600077 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95825889-a81f-4738-a105-1ff28b59e21f-console-oauth-config\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600118 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600119 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95825889-a81f-4738-a105-1ff28b59e21f-tmp\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600134 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-trusted-ca-bundle\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600162 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-service-ca\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600196 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6tw\" (UniqueName: \"kubernetes.io/projected/95825889-a81f-4738-a105-1ff28b59e21f-kube-api-access-wf6tw\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600239 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-oauth-serving-cert\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600268 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-console-config\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600310 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95825889-a81f-4738-a105-1ff28b59e21f-console-serving-cert\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600362 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw2xr\" (UniqueName: \"kubernetes.io/projected/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-kube-api-access-hw2xr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:05.600378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600376 4625 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:05.600807 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600394 4625 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32-console-oauth-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:05.600807 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600531 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95825889-a81f-4738-a105-1ff28b59e21f-tmp\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.600973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.600948 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-service-ca\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.601110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.601088 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-console-config\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.601170 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.601149 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-trusted-ca-bundle\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.601352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.601335 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95825889-a81f-4738-a105-1ff28b59e21f-oauth-serving-cert\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.602694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.602672 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95825889-a81f-4738-a105-1ff28b59e21f-console-serving-cert\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.602800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.602700 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95825889-a81f-4738-a105-1ff28b59e21f-console-oauth-config\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.622562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.622536 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6tw\" (UniqueName: \"kubernetes.io/projected/95825889-a81f-4738-a105-1ff28b59e21f-kube-api-access-wf6tw\") pod \"console-5cb57fd44f-zqgjf\" (UID: \"95825889-a81f-4738-a105-1ff28b59e21f\") " pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:05.919216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:05.919131 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:06.072151 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.072119 4625 generic.go:356] "Generic (PLEG): container finished" podID="b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" containerID="edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe" exitCode=0 Nov 03 22:14:06.072151 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.072156 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f7df997d-hgx6n" event={"ID":"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32","Type":"ContainerDied","Data":"edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe"} Nov 03 22:14:06.072651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.072168 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f7df997d-hgx6n" Nov 03 22:14:06.072651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.072195 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f7df997d-hgx6n" event={"ID":"b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32","Type":"ContainerDied","Data":"6e39bea3612e8933c0e56b52d6e403fd0889d52ff4528d4162627f6982234289"} Nov 03 22:14:06.072651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.072216 4625 scope.go:122] "RemoveContainer" containerID="edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe" Nov 03 22:14:06.089104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.089085 4625 scope.go:122] "RemoveContainer" containerID="edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe" Nov 03 22:14:06.089430 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:14:06.089409 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe\": container with ID starting with edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe not found: ID does not exist" containerID="edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe" Nov 03 22:14:06.089502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.089441 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe"} err="failed to get container status \"edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe\": rpc error: code = NotFound desc = could not find container \"edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe\": container with ID starting with edba09cf96c46a3d6e427f24bd18b8a2794372e0be524a8cba2df9cdac3e5cfe not found: ID does not exist" Nov 03 22:14:06.121164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.121130 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f7df997d-hgx6n"] Nov 03 22:14:06.126300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.126278 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f7df997d-hgx6n"] Nov 03 22:14:06.218062 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.217967 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cb57fd44f-zqgjf"] Nov 03 22:14:06.222944 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:14:06.222918 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95825889_a81f_4738_a105_1ff28b59e21f.slice/crio-fd4909ee8c924126d189bd3066a2ce921d87b5483bc54cfa514bd69cd2c3fa91 WatchSource:0}: Error finding container fd4909ee8c924126d189bd3066a2ce921d87b5483bc54cfa514bd69cd2c3fa91: Status 404 returned error can't find the container with id fd4909ee8c924126d189bd3066a2ce921d87b5483bc54cfa514bd69cd2c3fa91 Nov 03 22:14:06.728191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:06.728156 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" path="/var/lib/kubelet/pods/b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32/volumes" Nov 03 22:14:07.077560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:07.077452 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb57fd44f-zqgjf" event={"ID":"95825889-a81f-4738-a105-1ff28b59e21f","Type":"ContainerStarted","Data":"30b2b2f886cb1d1692aba07dec7a9d2ccc3e6a72b0781359ff03edb2104d3d2c"} Nov 03 22:14:07.077560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:07.077516 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb57fd44f-zqgjf" event={"ID":"95825889-a81f-4738-a105-1ff28b59e21f","Type":"ContainerStarted","Data":"fd4909ee8c924126d189bd3066a2ce921d87b5483bc54cfa514bd69cd2c3fa91"} Nov 03 22:14:07.118380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:07.118324 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5cb57fd44f-zqgjf" podStartSLOduration=2.118310058 podStartE2EDuration="2.118310058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:07.117238203 +0000 UTC m=+68.546574196" watchObservedRunningTime="2026-11-03 22:14:07.118310058 +0000 UTC m=+68.547646052" Nov 03 22:14:15.919532 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:15.919492 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:15.920041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:15.919573 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:15.924277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:15.924255 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:16.116639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:16.116608 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cb57fd44f-zqgjf" Nov 03 22:14:16.188215 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:16.188136 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5895c66dc9-bh25s"] Nov 03 22:14:32.184455 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:14:32.184414 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:14:36.180608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:36.180530 4625 generic.go:356] "Generic (PLEG): container finished" podID="58c6431b-8263-435c-8461-74365d4f9ce6" containerID="d8a3f2988d239f3e10d0174d5e4bd60adcf196f6fccef5839b096235e133fa25" exitCode=0 Nov 03 22:14:36.180608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:36.180569 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" event={"ID":"58c6431b-8263-435c-8461-74365d4f9ce6","Type":"ContainerDied","Data":"d8a3f2988d239f3e10d0174d5e4bd60adcf196f6fccef5839b096235e133fa25"} Nov 03 22:14:37.317692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.317666 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:14:37.444586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.444489 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-service-ca-bundle\") pod \"58c6431b-8263-435c-8461-74365d4f9ce6\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " Nov 03 22:14:37.444586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.444559 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-trusted-ca-bundle\") pod \"58c6431b-8263-435c-8461-74365d4f9ce6\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " Nov 03 22:14:37.444826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.444609 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c6431b-8263-435c-8461-74365d4f9ce6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58c6431b-8263-435c-8461-74365d4f9ce6-archives-path\") pod \"58c6431b-8263-435c-8461-74365d4f9ce6\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " Nov 03 22:14:37.444826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.444671 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c6431b-8263-435c-8461-74365d4f9ce6-kube-api-access-hpj8c\" (UniqueName: \"kubernetes.io/projected/58c6431b-8263-435c-8461-74365d4f9ce6-kube-api-access-hpj8c\") pod \"58c6431b-8263-435c-8461-74365d4f9ce6\" (UID: \"58c6431b-8263-435c-8461-74365d4f9ce6\") " Nov 03 22:14:37.444974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.444947 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-service-ca-bundle" pod "58c6431b-8263-435c-8461-74365d4f9ce6" (UID: "58c6431b-8263-435c-8461-74365d4f9ce6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:37.445034 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.444977 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c6431b-8263-435c-8461-74365d4f9ce6-archives-path" pod "58c6431b-8263-435c-8461-74365d4f9ce6" (UID: "58c6431b-8263-435c-8461-74365d4f9ce6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:14:37.445034 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.445001 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-trusted-ca-bundle" pod "58c6431b-8263-435c-8461-74365d4f9ce6" (UID: "58c6431b-8263-435c-8461-74365d4f9ce6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:37.446856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.446831 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c6431b-8263-435c-8461-74365d4f9ce6-kube-api-access-hpj8c" pod "58c6431b-8263-435c-8461-74365d4f9ce6" (UID: "58c6431b-8263-435c-8461-74365d4f9ce6"). InnerVolumeSpecName "kube-api-access-hpj8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:37.545692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.545646 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpj8c\" (UniqueName: \"kubernetes.io/projected/58c6431b-8263-435c-8461-74365d4f9ce6-kube-api-access-hpj8c\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:37.545692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.545684 4625 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-service-ca-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:37.545692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.545696 4625 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58c6431b-8263-435c-8461-74365d4f9ce6-trusted-ca-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:37.545959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:37.545708 4625 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58c6431b-8263-435c-8461-74365d4f9ce6-archives-path\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:38.188688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:38.188650 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" event={"ID":"58c6431b-8263-435c-8461-74365d4f9ce6","Type":"ContainerDied","Data":"7ab5a7b615b98a8c418babf0b2a381fb4c9f04a25f5c96f65846a173f1840fbf"} Nov 03 22:14:38.188688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:38.188689 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmzk9-8dvsz" Nov 03 22:14:38.189000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:38.188704 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab5a7b615b98a8c418babf0b2a381fb4c9f04a25f5c96f65846a173f1840fbf" Nov 03 22:14:41.210670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.210610 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5895c66dc9-bh25s" podUID="86a88332-8fdd-4c0d-8f3e-4578d6932bb5" containerName="console" containerID="cri-o://74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10" gracePeriod=15 Nov 03 22:14:41.463099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.463040 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:14:41.582361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582318 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-trusted-ca-bundle\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582369 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-tmp\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582403 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-oauth-serving-cert\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582448 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-oauth-config\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582503 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-serving-cert\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582551 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-kube-api-access-slb4x\" (UniqueName: \"kubernetes.io/projected/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-kube-api-access-slb4x\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582584 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-config\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-config\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582609 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-service-ca\") pod \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\" (UID: \"86a88332-8fdd-4c0d-8f3e-4578d6932bb5\") " Nov 03 22:14:41.582855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582674 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-tmp" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:14:41.582952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582862 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-trusted-ca-bundle" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:41.582952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582895 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-oauth-serving-cert" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:41.583061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.582995 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-config" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:41.583121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.583082 4625 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-trusted-ca-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:41.583230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.583212 4625 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-tmp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:41.583276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.583235 4625 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-oauth-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:41.583276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.583252 4625 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:41.583276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.583262 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-service-ca" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:41.584991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.584955 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-serving-cert" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:41.585308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.585283 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-oauth-config" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:41.585308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.585283 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-kube-api-access-slb4x" pod "86a88332-8fdd-4c0d-8f3e-4578d6932bb5" (UID: "86a88332-8fdd-4c0d-8f3e-4578d6932bb5"). InnerVolumeSpecName "kube-api-access-slb4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:41.683633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.683595 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slb4x\" (UniqueName: \"kubernetes.io/projected/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-kube-api-access-slb4x\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:41.683633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.683627 4625 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:41.683633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.683637 4625 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-oauth-config\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:41.683894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:41.683646 4625 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86a88332-8fdd-4c0d-8f3e-4578d6932bb5-console-serving-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:14:42.203921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.203889 4625 generic.go:356] "Generic (PLEG): container finished" podID="86a88332-8fdd-4c0d-8f3e-4578d6932bb5" containerID="74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10" exitCode=0 Nov 03 22:14:42.204098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.203937 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895c66dc9-bh25s" Nov 03 22:14:42.204098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.203966 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895c66dc9-bh25s" event={"ID":"86a88332-8fdd-4c0d-8f3e-4578d6932bb5","Type":"ContainerDied","Data":"74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10"} Nov 03 22:14:42.204098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.204004 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895c66dc9-bh25s" event={"ID":"86a88332-8fdd-4c0d-8f3e-4578d6932bb5","Type":"ContainerDied","Data":"bf0b991b1a909b3fe98421ee23c3b8842c1ba49dd4184cd3f5e3b09640633ffb"} Nov 03 22:14:42.204098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.204023 4625 scope.go:122] "RemoveContainer" containerID="74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10" Nov 03 22:14:42.212665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.212458 4625 scope.go:122] "RemoveContainer" containerID="74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10" Nov 03 22:14:42.212931 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:14:42.212767 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10\": container with ID starting with 74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10 not found: ID does not exist" containerID="74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10" Nov 03 22:14:42.212931 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.212788 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10"} err="failed to get container status \"74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10\": rpc error: code = NotFound desc = could not find container \"74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10\": container with ID starting with 74a5069af36839bec61e2ff4e6098a070754c358ef0851b721fec9259cfcba10 not found: ID does not exist" Nov 03 22:14:42.251893 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.251863 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5895c66dc9-bh25s"] Nov 03 22:14:42.262929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.262900 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5895c66dc9-bh25s"] Nov 03 22:14:42.724755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:14:42.724704 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a88332-8fdd-4c0d-8f3e-4578d6932bb5" path="/var/lib/kubelet/pods/86a88332-8fdd-4c0d-8f3e-4578d6932bb5/volumes" Nov 03 22:15:02.183850 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:15:02.183808 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:15:32.183940 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:15:32.183855 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:16:02.183587 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:16:02.183551 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:16:20.776680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.776642 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx"] Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777043 4625 cpu_manager.go:414] "containerMap: removing container" podUID="58c6431b-8263-435c-8461-74365d4f9ce6" containerName="insights-gathering" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777059 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c6431b-8263-435c-8461-74365d4f9ce6" containerName="insights-gathering" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777069 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" containerName="console" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777075 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" containerName="console" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777093 4625 cpu_manager.go:414] "containerMap: removing container" podUID="86a88332-8fdd-4c0d-8f3e-4578d6932bb5" containerName="console" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777099 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a88332-8fdd-4c0d-8f3e-4578d6932bb5" containerName="console" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777151 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c6431b-8263-435c-8461-74365d4f9ce6" containerName="insights-gathering" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777159 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6dbb8a3-be8d-4560-9d4f-b6223ff7ab32" containerName="console" Nov 03 22:16:20.777326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.777167 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a88332-8fdd-4c0d-8f3e-4578d6932bb5" containerName="console" Nov 03 22:16:20.780102 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.780085 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:20.794507 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.794482 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 22:16:20.794636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.794563 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nv5dh\"" Nov 03 22:16:20.805971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.805939 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 22:16:20.807217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.807182 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx"] Nov 03 22:16:20.919079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.919041 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rkf\" (UniqueName: \"kubernetes.io/projected/26884d2c-6314-4cf8-b455-43dedd245166-kube-api-access-t5rkf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:20.919266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.919089 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:20.919266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.919168 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:20.946971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.946930 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr"] Nov 03 22:16:20.950338 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.950317 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:20.971282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.971252 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-cg24t\"" Nov 03 22:16:20.972933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:20.972868 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr"] Nov 03 22:16:21.020217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020182 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.020399 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020229 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6l9p\" (UniqueName: \"kubernetes.io/projected/34d63026-0e16-45cf-be3c-325b7a00cb98-kube-api-access-n6l9p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.020399 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020307 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rkf\" (UniqueName: \"kubernetes.io/projected/26884d2c-6314-4cf8-b455-43dedd245166-kube-api-access-t5rkf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:21.020399 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020352 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:21.020522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020398 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:21.020522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020454 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.020752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020711 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:21.020803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.020756 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:21.045802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.045736 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rkf\" (UniqueName: \"kubernetes.io/projected/26884d2c-6314-4cf8-b455-43dedd245166-kube-api-access-t5rkf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:21.095391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.095345 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:21.121726 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.121687 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.121899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.121767 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.121899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.121811 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6l9p\" (UniqueName: \"kubernetes.io/projected/34d63026-0e16-45cf-be3c-325b7a00cb98-kube-api-access-n6l9p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.122154 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.122128 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.122223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.122137 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.145108 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.145073 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6l9p\" (UniqueName: \"kubernetes.io/projected/34d63026-0e16-45cf-be3c-325b7a00cb98-kube-api-access-n6l9p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.266732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.266681 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:21.402250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.402207 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx"] Nov 03 22:16:21.406639 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:21.406611 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26884d2c_6314_4cf8_b455_43dedd245166.slice/crio-ad8c5be86e905eef384250e214c561ef6ef5b2b6ead7be874cac5aad4b7832d5 WatchSource:0}: Error finding container ad8c5be86e905eef384250e214c561ef6ef5b2b6ead7be874cac5aad4b7832d5: Status 404 returned error can't find the container with id ad8c5be86e905eef384250e214c561ef6ef5b2b6ead7be874cac5aad4b7832d5 Nov 03 22:16:21.544369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.544337 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" event={"ID":"26884d2c-6314-4cf8-b455-43dedd245166","Type":"ContainerStarted","Data":"ad8c5be86e905eef384250e214c561ef6ef5b2b6ead7be874cac5aad4b7832d5"} Nov 03 22:16:21.568574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:21.568493 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr"] Nov 03 22:16:21.572322 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:21.572289 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d63026_0e16_45cf_be3c_325b7a00cb98.slice/crio-45a5b8151dbdc6f7d335e448c3df800e8162934deb5778dc2afd907c305c0e1a WatchSource:0}: Error finding container 45a5b8151dbdc6f7d335e448c3df800e8162934deb5778dc2afd907c305c0e1a: Status 404 returned error can't find the container with id 45a5b8151dbdc6f7d335e448c3df800e8162934deb5778dc2afd907c305c0e1a Nov 03 22:16:22.550519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:22.550414 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" event={"ID":"34d63026-0e16-45cf-be3c-325b7a00cb98","Type":"ContainerStarted","Data":"45a5b8151dbdc6f7d335e448c3df800e8162934deb5778dc2afd907c305c0e1a"} Nov 03 22:16:23.869769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:23.869736 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt"] Nov 03 22:16:23.873677 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:23.873650 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:23.899950 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:23.899915 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-j4ckn\"" Nov 03 22:16:23.901161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:23.901125 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt"] Nov 03 22:16:23.952731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:23.952684 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:23.952922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:23.952804 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:23.952922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:23.952843 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjks\" (UniqueName: \"kubernetes.io/projected/76f40792-fcce-43b0-95bc-5c984392888f-kube-api-access-7rjks\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.053847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.053809 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.053847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.053849 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.054045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.053877 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjks\" (UniqueName: \"kubernetes.io/projected/76f40792-fcce-43b0-95bc-5c984392888f-kube-api-access-7rjks\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.054225 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.054202 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.054271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.054234 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.076457 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.076431 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjks\" (UniqueName: \"kubernetes.io/projected/76f40792-fcce-43b0-95bc-5c984392888f-kube-api-access-7rjks\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.193460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.193377 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:24.507113 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.507038 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt"] Nov 03 22:16:24.511096 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:24.511052 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f40792_fcce_43b0_95bc_5c984392888f.slice/crio-c4eba0b12c9d4645a12064b13ff8c302e2dae301442de4585bb2d85903c8449a WatchSource:0}: Error finding container c4eba0b12c9d4645a12064b13ff8c302e2dae301442de4585bb2d85903c8449a: Status 404 returned error can't find the container with id c4eba0b12c9d4645a12064b13ff8c302e2dae301442de4585bb2d85903c8449a Nov 03 22:16:24.561204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.561164 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" event={"ID":"76f40792-fcce-43b0-95bc-5c984392888f","Type":"ContainerStarted","Data":"c4eba0b12c9d4645a12064b13ff8c302e2dae301442de4585bb2d85903c8449a"} Nov 03 22:16:24.681537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.681500 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb"] Nov 03 22:16:24.686445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.686418 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.703183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.703104 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb"] Nov 03 22:16:24.706943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.706901 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-tg49z\"" Nov 03 22:16:24.759923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.759832 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.760224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.759964 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.760224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.760029 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lxz\" (UniqueName: \"kubernetes.io/projected/f7470de4-2187-46d6-9e08-9fc604e23039-kube-api-access-75lxz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.778674 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.778635 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4"] Nov 03 22:16:24.783142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.783111 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.801273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.801245 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-sdbqf\"" Nov 03 22:16:24.803633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.803609 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4"] Nov 03 22:16:24.860686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.860646 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv6pr\" (UniqueName: \"kubernetes.io/projected/3dd1c3de-b95b-4e82-9e9b-908e166430fa-kube-api-access-qv6pr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.860886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.860707 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.860886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.860857 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.861020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.860903 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.861020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.860952 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lxz\" (UniqueName: \"kubernetes.io/projected/f7470de4-2187-46d6-9e08-9fc604e23039-kube-api-access-75lxz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.861020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.860992 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.861491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.861470 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.861617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.861551 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.880962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.880930 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh"] Nov 03 22:16:24.885020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.885000 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:24.887006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.886985 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lxz\" (UniqueName: \"kubernetes.io/projected/f7470de4-2187-46d6-9e08-9fc604e23039-kube-api-access-75lxz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:24.901813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.901789 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pjmg6\"" Nov 03 22:16:24.903698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.903674 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh"] Nov 03 22:16:24.962094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962055 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcsq\" (UniqueName: \"kubernetes.io/projected/06724f69-0c61-4139-a1fa-4628f6b8267b-kube-api-access-8fcsq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:24.962295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962117 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.962295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962213 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.962295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962282 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:24.962456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962312 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:24.962456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962345 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv6pr\" (UniqueName: \"kubernetes.io/projected/3dd1c3de-b95b-4e82-9e9b-908e166430fa-kube-api-access-qv6pr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.962541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962524 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.962589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.962555 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.975175 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.975141 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx"] Nov 03 22:16:24.984745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.982607 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:24.989366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.989335 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv6pr\" (UniqueName: \"kubernetes.io/projected/3dd1c3de-b95b-4e82-9e9b-908e166430fa-kube-api-access-qv6pr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:24.998184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.998158 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx"] Nov 03 22:16:24.998437 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:24.998418 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-tw8hn\"" Nov 03 22:16:25.009700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.009682 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:25.064067 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.064026 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:25.064228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.064076 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:25.064228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.064142 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.064228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.064182 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcsq\" (UniqueName: \"kubernetes.io/projected/06724f69-0c61-4139-a1fa-4628f6b8267b-kube-api-access-8fcsq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:25.064403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.064227 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.064403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.064261 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwrb2\" (UniqueName: \"kubernetes.io/projected/af1176bd-e222-4009-9328-059825ea65a8-kube-api-access-gwrb2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.064776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.064756 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:25.065028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.065011 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:25.089283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.089198 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcsq\" (UniqueName: \"kubernetes.io/projected/06724f69-0c61-4139-a1fa-4628f6b8267b-kube-api-access-8fcsq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:25.104197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.103914 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:25.166004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.165362 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.166004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.165525 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.166004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.165562 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwrb2\" (UniqueName: \"kubernetes.io/projected/af1176bd-e222-4009-9328-059825ea65a8-kube-api-access-gwrb2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.166004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.165858 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.166369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.166337 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.196166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.196140 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwrb2\" (UniqueName: \"kubernetes.io/projected/af1176bd-e222-4009-9328-059825ea65a8-kube-api-access-gwrb2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.217309 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.217280 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:25.307585 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.307352 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:25.323603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.323570 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb"] Nov 03 22:16:25.327462 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:25.327436 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7470de4_2187_46d6_9e08_9fc604e23039.slice/crio-ef2168d1b825b9e54f413a71db3ce4950a0f799ac7164882f401ad1d09639d21 WatchSource:0}: Error finding container ef2168d1b825b9e54f413a71db3ce4950a0f799ac7164882f401ad1d09639d21: Status 404 returned error can't find the container with id ef2168d1b825b9e54f413a71db3ce4950a0f799ac7164882f401ad1d09639d21 Nov 03 22:16:25.427216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.427135 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4"] Nov 03 22:16:25.432349 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:25.432317 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd1c3de_b95b_4e82_9e9b_908e166430fa.slice/crio-d844b4352bc791566994b6c27e5146b51e884bcde00e8571fb11e2243d1c9043 WatchSource:0}: Error finding container d844b4352bc791566994b6c27e5146b51e884bcde00e8571fb11e2243d1c9043: Status 404 returned error can't find the container with id d844b4352bc791566994b6c27e5146b51e884bcde00e8571fb11e2243d1c9043 Nov 03 22:16:25.533080 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.532996 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh"] Nov 03 22:16:25.536829 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:25.536796 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06724f69_0c61_4139_a1fa_4628f6b8267b.slice/crio-99fd08cd967b8d1f00a7a237dbbeee7e688565a832e4b9c85b1be331ec7b55ea WatchSource:0}: Error finding container 99fd08cd967b8d1f00a7a237dbbeee7e688565a832e4b9c85b1be331ec7b55ea: Status 404 returned error can't find the container with id 99fd08cd967b8d1f00a7a237dbbeee7e688565a832e4b9c85b1be331ec7b55ea Nov 03 22:16:25.568841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.568759 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" event={"ID":"3dd1c3de-b95b-4e82-9e9b-908e166430fa","Type":"ContainerStarted","Data":"d844b4352bc791566994b6c27e5146b51e884bcde00e8571fb11e2243d1c9043"} Nov 03 22:16:25.570654 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.570596 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" event={"ID":"f7470de4-2187-46d6-9e08-9fc604e23039","Type":"ContainerStarted","Data":"ef2168d1b825b9e54f413a71db3ce4950a0f799ac7164882f401ad1d09639d21"} Nov 03 22:16:25.574619 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.574594 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" event={"ID":"06724f69-0c61-4139-a1fa-4628f6b8267b","Type":"ContainerStarted","Data":"99fd08cd967b8d1f00a7a237dbbeee7e688565a832e4b9c85b1be331ec7b55ea"} Nov 03 22:16:25.622147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:25.622114 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx"] Nov 03 22:16:25.627796 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:25.627765 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1176bd_e222_4009_9328_059825ea65a8.slice/crio-b9efdcb45df11d688c09f67557b24e7cab6dbece00c211d297c16295fb625699 WatchSource:0}: Error finding container b9efdcb45df11d688c09f67557b24e7cab6dbece00c211d297c16295fb625699: Status 404 returned error can't find the container with id b9efdcb45df11d688c09f67557b24e7cab6dbece00c211d297c16295fb625699 Nov 03 22:16:26.586271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:26.586167 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" event={"ID":"af1176bd-e222-4009-9328-059825ea65a8","Type":"ContainerStarted","Data":"b9efdcb45df11d688c09f67557b24e7cab6dbece00c211d297c16295fb625699"} Nov 03 22:16:26.831792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:26.831755 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f"] Nov 03 22:16:26.836873 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:26.836232 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:26.871199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:26.871117 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f"] Nov 03 22:16:26.982468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:26.982406 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:26.982808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:26.982495 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:26.982808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:26.982545 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbjs\" (UniqueName: \"kubernetes.io/projected/82ecdb0a-033a-4433-9491-7478d3494b88-kube-api-access-hwbjs\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.083211 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.083168 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.083471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.083242 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.083471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.083291 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbjs\" (UniqueName: \"kubernetes.io/projected/82ecdb0a-033a-4433-9491-7478d3494b88-kube-api-access-hwbjs\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.084128 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.084098 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.084403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.084382 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.118639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.117914 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbjs\" (UniqueName: \"kubernetes.io/projected/82ecdb0a-033a-4433-9491-7478d3494b88-kube-api-access-hwbjs\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.159336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.159062 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:27.207334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.205191 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd"] Nov 03 22:16:27.211445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.211404 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.228092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.227398 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd"] Nov 03 22:16:27.273211 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.271426 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p"] Nov 03 22:16:27.275740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.275499 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.285234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.285213 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b256g\" (UniqueName: \"kubernetes.io/projected/f6d37266-e999-4a6c-9451-c1dfec5d62b0-kube-api-access-b256g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.285780 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.285758 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.285952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.285937 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.297078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.296767 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9d7jc\"" Nov 03 22:16:27.297078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.296905 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:16:27.297078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.296988 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:16:27.297078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.297048 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p"] Nov 03 22:16:27.297363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.297054 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387169 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387221 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387250 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387295 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387326 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387353 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387393 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387421 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzr4\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-kube-api-access-nlzr4\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387463 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387496 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.387537 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b256g\" (UniqueName: \"kubernetes.io/projected/f6d37266-e999-4a6c-9451-c1dfec5d62b0-kube-api-access-b256g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.388618 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.389141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.389057 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.391833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.391686 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.402164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.400460 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs"] Nov 03 22:16:27.405527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.404581 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.413693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.413636 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b256g\" (UniqueName: \"kubernetes.io/projected/f6d37266-e999-4a6c-9451-c1dfec5d62b0-kube-api-access-b256g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.420037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.420012 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-bjfj4\"" Nov 03 22:16:27.426624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.426555 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs"] Nov 03 22:16:27.492664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492624 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.492859 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492676 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.492859 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492816 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqqw\" (UniqueName: \"kubernetes.io/projected/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-kube-api-access-sbqqw\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.492859 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492838 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.493020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492887 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.493020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492911 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.493020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492933 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.493020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492974 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.493020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.492989 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.493020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493008 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493061 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493102 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493142 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493170 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493197 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493213 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzr4\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-kube-api-access-nlzr4\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493234 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.493263 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.494095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.494070 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.495345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.495323 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.495584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.495563 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.496206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.496183 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.496361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.496344 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.516287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.516253 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.517017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.516991 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f"] Nov 03 22:16:27.520156 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.520020 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.521799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.521776 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzr4\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-kube-api-access-nlzr4\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.539219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.539197 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:27.546088 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.546032 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-v828p\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.593693 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.593774 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.593818 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.593849 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.593895 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.593954 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.594004 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.594042 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqqw\" (UniqueName: \"kubernetes.io/projected/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-kube-api-access-sbqqw\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.594061 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.594555 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.595255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.594859 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.599396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.597097 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.599396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.597609 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.599396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.597763 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:27.599396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.597799 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.599396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.598377 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.599396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.598568 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" event={"ID":"82ecdb0a-033a-4433-9491-7478d3494b88","Type":"ContainerStarted","Data":"cd568d079143845f6d684ed44f3c85336946ce3019c131bf4e02a6e6b5143f95"} Nov 03 22:16:27.600320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.600184 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.628469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.628405 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqqw\" (UniqueName: \"kubernetes.io/projected/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-kube-api-access-sbqqw\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.629592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.629546 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7f55fa-0e84-4ffc-9601-6c1c53c1b669-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs\" (UID: \"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.733406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.733369 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:27.915960 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.915700 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd"] Nov 03 22:16:27.922379 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:27.922341 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d37266_e999_4a6c_9451_c1dfec5d62b0.slice/crio-6dd556b11c855bf2e79768aa1ae3bb47026446d05a120a63feba49d341ea427c WatchSource:0}: Error finding container 6dd556b11c855bf2e79768aa1ae3bb47026446d05a120a63feba49d341ea427c: Status 404 returned error can't find the container with id 6dd556b11c855bf2e79768aa1ae3bb47026446d05a120a63feba49d341ea427c Nov 03 22:16:27.961278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:27.960306 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p"] Nov 03 22:16:27.967087 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:27.967046 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82444d7_3c98_4ed4_8ee9_3e227757dbed.slice/crio-77f803923557efc7787fa6a0fbac3ed0d3aa71029d0684cc0d05dd0fa5c77edd WatchSource:0}: Error finding container 77f803923557efc7787fa6a0fbac3ed0d3aa71029d0684cc0d05dd0fa5c77edd: Status 404 returned error can't find the container with id 77f803923557efc7787fa6a0fbac3ed0d3aa71029d0684cc0d05dd0fa5c77edd Nov 03 22:16:28.071588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:28.071497 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs"] Nov 03 22:16:28.087191 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:16:28.087156 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7f55fa_0e84_4ffc_9601_6c1c53c1b669.slice/crio-cd2375df1a47cbd9942f39f567eab5d435a112cf8cd6a3bf9fd4f6519f6f6009 WatchSource:0}: Error finding container cd2375df1a47cbd9942f39f567eab5d435a112cf8cd6a3bf9fd4f6519f6f6009: Status 404 returned error can't find the container with id cd2375df1a47cbd9942f39f567eab5d435a112cf8cd6a3bf9fd4f6519f6f6009 Nov 03 22:16:28.609537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:28.609496 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" event={"ID":"f6d37266-e999-4a6c-9451-c1dfec5d62b0","Type":"ContainerStarted","Data":"6dd556b11c855bf2e79768aa1ae3bb47026446d05a120a63feba49d341ea427c"} Nov 03 22:16:28.614801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:28.614764 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" event={"ID":"d82444d7-3c98-4ed4-8ee9-3e227757dbed","Type":"ContainerStarted","Data":"77f803923557efc7787fa6a0fbac3ed0d3aa71029d0684cc0d05dd0fa5c77edd"} Nov 03 22:16:28.623320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:28.623235 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" event={"ID":"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669","Type":"ContainerStarted","Data":"cd2375df1a47cbd9942f39f567eab5d435a112cf8cd6a3bf9fd4f6519f6f6009"} Nov 03 22:16:32.245830 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:16:32.245732 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:16:35.700171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:35.700131 4625 generic.go:356] "Generic (PLEG): container finished" podID="f7470de4-2187-46d6-9e08-9fc604e23039" containerID="ff1f2cbcc6c8a74dc665e20c5454e909b929183da9df9579db565e98b93a11f7" exitCode=0 Nov 03 22:16:35.700698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:35.700232 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" event={"ID":"f7470de4-2187-46d6-9e08-9fc604e23039","Type":"ContainerDied","Data":"ff1f2cbcc6c8a74dc665e20c5454e909b929183da9df9579db565e98b93a11f7"} Nov 03 22:16:35.708003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:35.707971 4625 generic.go:356] "Generic (PLEG): container finished" podID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerID="118fea293ba1806d497c9e2fdc8c3bc3fe8c043bbea53e1a777e1f1a57d44d37" exitCode=0 Nov 03 22:16:35.708121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:35.708105 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" event={"ID":"34d63026-0e16-45cf-be3c-325b7a00cb98","Type":"ContainerDied","Data":"118fea293ba1806d497c9e2fdc8c3bc3fe8c043bbea53e1a777e1f1a57d44d37"} Nov 03 22:16:35.715300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:35.715273 4625 generic.go:356] "Generic (PLEG): container finished" podID="26884d2c-6314-4cf8-b455-43dedd245166" containerID="a1f5d67d12cbddd8b18f856bb8dd57847ef00045260394013d9918c6f5a9c2ca" exitCode=0 Nov 03 22:16:35.715412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:35.715333 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" event={"ID":"26884d2c-6314-4cf8-b455-43dedd245166","Type":"ContainerDied","Data":"a1f5d67d12cbddd8b18f856bb8dd57847ef00045260394013d9918c6f5a9c2ca"} Nov 03 22:16:36.461357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.461322 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:16:36.461468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.461397 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:16:36.461468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.461432 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:16:36.730922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.730893 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" event={"ID":"d82444d7-3c98-4ed4-8ee9-3e227757dbed","Type":"ContainerStarted","Data":"821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7"} Nov 03 22:16:36.742226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.742173 4625 generic.go:356] "Generic (PLEG): container finished" podID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerID="59b4ec419894faf67a8b9cb22a8f60cec41afdb26f9d653a5aabbb1f9487130a" exitCode=0 Nov 03 22:16:36.742493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.742470 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" event={"ID":"34d63026-0e16-45cf-be3c-325b7a00cb98","Type":"ContainerDied","Data":"59b4ec419894faf67a8b9cb22a8f60cec41afdb26f9d653a5aabbb1f9487130a"} Nov 03 22:16:36.745225 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.745203 4625 generic.go:356] "Generic (PLEG): container finished" podID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerID="5303a56d4dffed5633d96da76d1812a5451b9392b3a19a310da0c357c1477c3f" exitCode=0 Nov 03 22:16:36.745442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.745426 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" event={"ID":"06724f69-0c61-4139-a1fa-4628f6b8267b","Type":"ContainerDied","Data":"5303a56d4dffed5633d96da76d1812a5451b9392b3a19a310da0c357c1477c3f"} Nov 03 22:16:36.748108 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.748089 4625 generic.go:356] "Generic (PLEG): container finished" podID="af1176bd-e222-4009-9328-059825ea65a8" containerID="52b5161d8b72c85362dc121f3f97054f8bcbf86f4c2eb04d209a513d41a310dd" exitCode=0 Nov 03 22:16:36.748182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.748167 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" event={"ID":"af1176bd-e222-4009-9328-059825ea65a8","Type":"ContainerDied","Data":"52b5161d8b72c85362dc121f3f97054f8bcbf86f4c2eb04d209a513d41a310dd"} Nov 03 22:16:36.753809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.753198 4625 generic.go:356] "Generic (PLEG): container finished" podID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerID="fc56a990736d28c5960df6a2775548199ce51dccae011174197e8d5eda8f6373" exitCode=0 Nov 03 22:16:36.753809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.753231 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" event={"ID":"3dd1c3de-b95b-4e82-9e9b-908e166430fa","Type":"ContainerDied","Data":"fc56a990736d28c5960df6a2775548199ce51dccae011174197e8d5eda8f6373"} Nov 03 22:16:36.776370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:36.774965 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podStartSLOduration=1.285655419 podStartE2EDuration="9.77494844s" totalImagesPullingTime="8.489293021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:36.774175299 +0000 UTC m=+218.203511292" watchObservedRunningTime="2026-11-03 22:16:36.77494844 +0000 UTC m=+218.204284434" Nov 03 22:16:37.599699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.599669 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:37.605231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.604924 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:37.763529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.763499 4625 generic.go:356] "Generic (PLEG): container finished" podID="f7470de4-2187-46d6-9e08-9fc604e23039" containerID="e29baad400352f5719f081d8b9cb65bf6465d183b953f1a012e4ed1a8e9010f4" exitCode=0 Nov 03 22:16:37.763922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.763624 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" event={"ID":"f7470de4-2187-46d6-9e08-9fc604e23039","Type":"ContainerDied","Data":"e29baad400352f5719f081d8b9cb65bf6465d183b953f1a012e4ed1a8e9010f4"} Nov 03 22:16:37.767086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.766929 4625 generic.go:356] "Generic (PLEG): container finished" podID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerID="27340cc1759186bd33dde15eebcba82aaa60313a44b13f9f19f97bf90542d452" exitCode=0 Nov 03 22:16:37.767086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.767034 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" event={"ID":"06724f69-0c61-4139-a1fa-4628f6b8267b","Type":"ContainerDied","Data":"27340cc1759186bd33dde15eebcba82aaa60313a44b13f9f19f97bf90542d452"} Nov 03 22:16:37.771187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.770136 4625 generic.go:356] "Generic (PLEG): container finished" podID="82ecdb0a-033a-4433-9491-7478d3494b88" containerID="69b0c7db80f975ced066f9632e71f69aa15b919231d912094e0b960908e08423" exitCode=0 Nov 03 22:16:37.771187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.770701 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" event={"ID":"82ecdb0a-033a-4433-9491-7478d3494b88","Type":"ContainerDied","Data":"69b0c7db80f975ced066f9632e71f69aa15b919231d912094e0b960908e08423"} Nov 03 22:16:37.771187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.771138 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:37.772304 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.772241 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:16:37.859652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.859176 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:16:37.859652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.859266 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:16:37.859652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:37.859316 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:16:38.779938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:38.779896 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" event={"ID":"6c7f55fa-0e84-4ffc-9601-6c1c53c1b669","Type":"ContainerStarted","Data":"1e41a5e36130b6b3992c1dd09cab4e577528d8ee17f30cc037fa1e0dcf0427c8"} Nov 03 22:16:38.783274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:38.783243 4625 generic.go:356] "Generic (PLEG): container finished" podID="af1176bd-e222-4009-9328-059825ea65a8" containerID="490745d6809d58847af4e545faf586d1530c0ef23ea0d7c44d95b87446829dc1" exitCode=0 Nov 03 22:16:38.783414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:38.783326 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" event={"ID":"af1176bd-e222-4009-9328-059825ea65a8","Type":"ContainerDied","Data":"490745d6809d58847af4e545faf586d1530c0ef23ea0d7c44d95b87446829dc1"} Nov 03 22:16:38.787762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:38.787230 4625 generic.go:356] "Generic (PLEG): container finished" podID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerID="5d52911476a62bc1886034e32bf9dee6c129241a56eac6261ee22cc2e1a60b8a" exitCode=0 Nov 03 22:16:38.789113 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:38.789062 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" event={"ID":"f6d37266-e999-4a6c-9451-c1dfec5d62b0","Type":"ContainerDied","Data":"5d52911476a62bc1886034e32bf9dee6c129241a56eac6261ee22cc2e1a60b8a"} Nov 03 22:16:38.847807 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:38.846253 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" podStartSLOduration=2.082172145 podStartE2EDuration="11.846234127s" totalImagesPullingTime="9.764061982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:38.834192278 +0000 UTC m=+220.263528274" watchObservedRunningTime="2026-11-03 22:16:38.846234127 +0000 UTC m=+220.275570121" Nov 03 22:16:39.734806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.734682 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:39.748439 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.740625 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:39.810538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.810512 4625 generic.go:356] "Generic (PLEG): container finished" podID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerID="d6a942531dd842961fd008f535e6bca2eea63876fca621560d3bfcf0c396c111" exitCode=0 Nov 03 22:16:39.810849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.810602 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" event={"ID":"f6d37266-e999-4a6c-9451-c1dfec5d62b0","Type":"ContainerDied","Data":"d6a942531dd842961fd008f535e6bca2eea63876fca621560d3bfcf0c396c111"} Nov 03 22:16:39.815877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.815845 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" event={"ID":"76f40792-fcce-43b0-95bc-5c984392888f","Type":"ContainerStarted","Data":"d11f817e15369ff47fa5bd85adfc1520d01772a3faf8a63402ede276950aea6e"} Nov 03 22:16:39.824423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.824372 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" event={"ID":"26884d2c-6314-4cf8-b455-43dedd245166","Type":"ContainerStarted","Data":"a17d64b3bd6884ceb79f7594e1d1de25181a1a30a3964c0818b1d14c4dd2484e"} Nov 03 22:16:39.825367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.825342 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:39.827371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:39.827349 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs" Nov 03 22:16:40.836256 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:40.836217 4625 generic.go:356] "Generic (PLEG): container finished" podID="26884d2c-6314-4cf8-b455-43dedd245166" containerID="a17d64b3bd6884ceb79f7594e1d1de25181a1a30a3964c0818b1d14c4dd2484e" exitCode=0 Nov 03 22:16:40.836740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:40.836354 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" event={"ID":"26884d2c-6314-4cf8-b455-43dedd245166","Type":"ContainerDied","Data":"a17d64b3bd6884ceb79f7594e1d1de25181a1a30a3964c0818b1d14c4dd2484e"} Nov 03 22:16:40.841479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:40.841426 4625 generic.go:356] "Generic (PLEG): container finished" podID="76f40792-fcce-43b0-95bc-5c984392888f" containerID="d11f817e15369ff47fa5bd85adfc1520d01772a3faf8a63402ede276950aea6e" exitCode=0 Nov 03 22:16:40.841606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:40.841501 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" event={"ID":"76f40792-fcce-43b0-95bc-5c984392888f","Type":"ContainerDied","Data":"d11f817e15369ff47fa5bd85adfc1520d01772a3faf8a63402ede276950aea6e"} Nov 03 22:16:40.849542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:40.849500 4625 generic.go:356] "Generic (PLEG): container finished" podID="82ecdb0a-033a-4433-9491-7478d3494b88" containerID="99890c23b8e65d5693db7be6a663420ef54092182c9b77d32f28755b559ac0e6" exitCode=0 Nov 03 22:16:40.849650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:40.849621 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" event={"ID":"82ecdb0a-033a-4433-9491-7478d3494b88","Type":"ContainerDied","Data":"99890c23b8e65d5693db7be6a663420ef54092182c9b77d32f28755b559ac0e6"} Nov 03 22:16:46.935516 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:46.934387 4625 generic.go:356] "Generic (PLEG): container finished" podID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerID="33f074a94b19431d000244661ab709eb97b39010b7acde443ea1b9acd1eb5506" exitCode=0 Nov 03 22:16:46.935516 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:46.934482 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" event={"ID":"3dd1c3de-b95b-4e82-9e9b-908e166430fa","Type":"ContainerDied","Data":"33f074a94b19431d000244661ab709eb97b39010b7acde443ea1b9acd1eb5506"} Nov 03 22:16:46.949568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:46.949534 4625 generic.go:356] "Generic (PLEG): container finished" podID="76f40792-fcce-43b0-95bc-5c984392888f" containerID="7e1d2ebb49bae73112b88d1aea6008f6c1a69498133eb0296e07b10a832d1a40" exitCode=0 Nov 03 22:16:46.949677 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:46.949600 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" event={"ID":"76f40792-fcce-43b0-95bc-5c984392888f","Type":"ContainerDied","Data":"7e1d2ebb49bae73112b88d1aea6008f6c1a69498133eb0296e07b10a832d1a40"} Nov 03 22:16:53.014293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:53.013642 4625 generic.go:356] "Generic (PLEG): container finished" podID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerID="7926f8b5ad37ef8d3acc3bdb02bd2438bffec78df5fc4fa9f978099ad16a9942" exitCode=0 Nov 03 22:16:53.014293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:53.013801 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" event={"ID":"34d63026-0e16-45cf-be3c-325b7a00cb98","Type":"ContainerDied","Data":"7926f8b5ad37ef8d3acc3bdb02bd2438bffec78df5fc4fa9f978099ad16a9942"} Nov 03 22:16:54.218277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.218233 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:54.268745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.268702 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-util\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-util\") pod \"34d63026-0e16-45cf-be3c-325b7a00cb98\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " Nov 03 22:16:54.268934 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.268793 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-bundle\") pod \"34d63026-0e16-45cf-be3c-325b7a00cb98\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " Nov 03 22:16:54.268934 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.268889 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d63026-0e16-45cf-be3c-325b7a00cb98-kube-api-access-n6l9p\" (UniqueName: \"kubernetes.io/projected/34d63026-0e16-45cf-be3c-325b7a00cb98-kube-api-access-n6l9p\") pod \"34d63026-0e16-45cf-be3c-325b7a00cb98\" (UID: \"34d63026-0e16-45cf-be3c-325b7a00cb98\") " Nov 03 22:16:54.271561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.271519 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-bundle" pod "34d63026-0e16-45cf-be3c-325b7a00cb98" (UID: "34d63026-0e16-45cf-be3c-325b7a00cb98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:54.274908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.274847 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d63026-0e16-45cf-be3c-325b7a00cb98-kube-api-access-n6l9p" pod "34d63026-0e16-45cf-be3c-325b7a00cb98" (UID: "34d63026-0e16-45cf-be3c-325b7a00cb98"). InnerVolumeSpecName "kube-api-access-n6l9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:54.275307 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.275285 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-util" pod "34d63026-0e16-45cf-be3c-325b7a00cb98" (UID: "34d63026-0e16-45cf-be3c-325b7a00cb98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:54.370608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.370566 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:54.370608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.370605 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6l9p\" (UniqueName: \"kubernetes.io/projected/34d63026-0e16-45cf-be3c-325b7a00cb98-kube-api-access-n6l9p\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:54.370871 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:54.370623 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d63026-0e16-45cf-be3c-325b7a00cb98-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:55.042667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:55.042574 4625 generic.go:356] "Generic (PLEG): container finished" podID="f7470de4-2187-46d6-9e08-9fc604e23039" containerID="f084835b4195706a3651d49229c843062381bb41064d6f854d67a531ca8a6a49" exitCode=0 Nov 03 22:16:55.043061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:55.042683 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" event={"ID":"f7470de4-2187-46d6-9e08-9fc604e23039","Type":"ContainerDied","Data":"f084835b4195706a3651d49229c843062381bb41064d6f854d67a531ca8a6a49"} Nov 03 22:16:55.047646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:55.047612 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" event={"ID":"34d63026-0e16-45cf-be3c-325b7a00cb98","Type":"ContainerDied","Data":"45a5b8151dbdc6f7d335e448c3df800e8162934deb5778dc2afd907c305c0e1a"} Nov 03 22:16:55.047788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:55.047672 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a5b8151dbdc6f7d335e448c3df800e8162934deb5778dc2afd907c305c0e1a" Nov 03 22:16:55.047788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:55.047775 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gb4dr" Nov 03 22:16:55.054041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:55.054009 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" event={"ID":"06724f69-0c61-4139-a1fa-4628f6b8267b","Type":"ContainerStarted","Data":"a0e6de7f60718289d43220079bea52874f1a92a7e47c2c91ee0e305eb9ce5454"} Nov 03 22:16:55.147498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:55.147235 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" podStartSLOduration=3.052696216 podStartE2EDuration="31.147213402s" totalImagesPullingTime="28.094517186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:55.144679991 +0000 UTC m=+236.574015986" watchObservedRunningTime="2026-11-03 22:16:55.147213402 +0000 UTC m=+236.576549396" Nov 03 22:16:56.077806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.077565 4625 generic.go:356] "Generic (PLEG): container finished" podID="af1176bd-e222-4009-9328-059825ea65a8" containerID="6816d33e2007de54580577f99ff89968ab2eefbae1c3199f413056e0908d0df3" exitCode=0 Nov 03 22:16:56.077806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.077740 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" event={"ID":"af1176bd-e222-4009-9328-059825ea65a8","Type":"ContainerDied","Data":"6816d33e2007de54580577f99ff89968ab2eefbae1c3199f413056e0908d0df3"} Nov 03 22:16:56.083793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.083757 4625 generic.go:356] "Generic (PLEG): container finished" podID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerID="a0e6de7f60718289d43220079bea52874f1a92a7e47c2c91ee0e305eb9ce5454" exitCode=0 Nov 03 22:16:56.084195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.084167 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" event={"ID":"06724f69-0c61-4139-a1fa-4628f6b8267b","Type":"ContainerDied","Data":"a0e6de7f60718289d43220079bea52874f1a92a7e47c2c91ee0e305eb9ce5454"} Nov 03 22:16:56.288909 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.288542 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:56.389287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.389252 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-bundle\") pod \"f7470de4-2187-46d6-9e08-9fc604e23039\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " Nov 03 22:16:56.389463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.389312 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7470de4-2187-46d6-9e08-9fc604e23039-kube-api-access-75lxz\" (UniqueName: \"kubernetes.io/projected/f7470de4-2187-46d6-9e08-9fc604e23039-kube-api-access-75lxz\") pod \"f7470de4-2187-46d6-9e08-9fc604e23039\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " Nov 03 22:16:56.389463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.389380 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-util\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-util\") pod \"f7470de4-2187-46d6-9e08-9fc604e23039\" (UID: \"f7470de4-2187-46d6-9e08-9fc604e23039\") " Nov 03 22:16:56.390285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.390256 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-bundle" pod "f7470de4-2187-46d6-9e08-9fc604e23039" (UID: "f7470de4-2187-46d6-9e08-9fc604e23039"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:56.402701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.400420 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7470de4-2187-46d6-9e08-9fc604e23039-kube-api-access-75lxz" pod "f7470de4-2187-46d6-9e08-9fc604e23039" (UID: "f7470de4-2187-46d6-9e08-9fc604e23039"). InnerVolumeSpecName "kube-api-access-75lxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:56.402701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.401297 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-util" pod "f7470de4-2187-46d6-9e08-9fc604e23039" (UID: "f7470de4-2187-46d6-9e08-9fc604e23039"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:56.490785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.490747 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:56.490785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.490789 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75lxz\" (UniqueName: \"kubernetes.io/projected/f7470de4-2187-46d6-9e08-9fc604e23039-kube-api-access-75lxz\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:56.491058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:56.490805 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7470de4-2187-46d6-9e08-9fc604e23039-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:57.093659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.093570 4625 generic.go:356] "Generic (PLEG): container finished" podID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerID="c44cd7fcac019a01ee45dc9c89bdb0ae0552785878cb196a923577da16c080de" exitCode=0 Nov 03 22:16:57.093659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.093653 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" event={"ID":"f6d37266-e999-4a6c-9451-c1dfec5d62b0","Type":"ContainerDied","Data":"c44cd7fcac019a01ee45dc9c89bdb0ae0552785878cb196a923577da16c080de"} Nov 03 22:16:57.096538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.096511 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" event={"ID":"3dd1c3de-b95b-4e82-9e9b-908e166430fa","Type":"ContainerStarted","Data":"945d5509d0b3b733c2f7d9c2bda7589a2c0165b941aec8f95b542f8cf6609abd"} Nov 03 22:16:57.100003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.099971 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" event={"ID":"76f40792-fcce-43b0-95bc-5c984392888f","Type":"ContainerStarted","Data":"a459df8cf06c031b8981fdc49c5aa9277d6153ce8ee5c4a1724dee3c207eecb2"} Nov 03 22:16:57.104190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.104170 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" Nov 03 22:16:57.104673 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.104654 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdvrb" event={"ID":"f7470de4-2187-46d6-9e08-9fc604e23039","Type":"ContainerDied","Data":"ef2168d1b825b9e54f413a71db3ce4950a0f799ac7164882f401ad1d09639d21"} Nov 03 22:16:57.104791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.104690 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef2168d1b825b9e54f413a71db3ce4950a0f799ac7164882f401ad1d09639d21" Nov 03 22:16:57.179683 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.179622 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" podStartSLOduration=2.733492395 podStartE2EDuration="34.179600544s" totalImagesPullingTime="31.446108149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:57.17795476 +0000 UTC m=+238.607290754" watchObservedRunningTime="2026-11-03 22:16:57.179600544 +0000 UTC m=+238.608936540" Nov 03 22:16:57.234565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.234141 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" podStartSLOduration=3.304807545 podStartE2EDuration="33.234122208s" totalImagesPullingTime="29.929314663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:57.232042785 +0000 UTC m=+238.661378777" watchObservedRunningTime="2026-11-03 22:16:57.234122208 +0000 UTC m=+238.663458202" Nov 03 22:16:57.239211 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.238940 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:57.297643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.296951 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-bundle\") pod \"06724f69-0c61-4139-a1fa-4628f6b8267b\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " Nov 03 22:16:57.297643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.297012 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06724f69-0c61-4139-a1fa-4628f6b8267b-kube-api-access-8fcsq\" (UniqueName: \"kubernetes.io/projected/06724f69-0c61-4139-a1fa-4628f6b8267b-kube-api-access-8fcsq\") pod \"06724f69-0c61-4139-a1fa-4628f6b8267b\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " Nov 03 22:16:57.297643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.297077 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-util\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-util\") pod \"06724f69-0c61-4139-a1fa-4628f6b8267b\" (UID: \"06724f69-0c61-4139-a1fa-4628f6b8267b\") " Nov 03 22:16:57.298130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.298084 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-bundle" pod "06724f69-0c61-4139-a1fa-4628f6b8267b" (UID: "06724f69-0c61-4139-a1fa-4628f6b8267b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:57.300054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.300026 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06724f69-0c61-4139-a1fa-4628f6b8267b-kube-api-access-8fcsq" pod "06724f69-0c61-4139-a1fa-4628f6b8267b" (UID: "06724f69-0c61-4139-a1fa-4628f6b8267b"). InnerVolumeSpecName "kube-api-access-8fcsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:57.303797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.303768 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-util" pod "06724f69-0c61-4139-a1fa-4628f6b8267b" (UID: "06724f69-0c61-4139-a1fa-4628f6b8267b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:57.345197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.345132 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:57.398593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.398558 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-bundle\") pod \"af1176bd-e222-4009-9328-059825ea65a8\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " Nov 03 22:16:57.398786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.398620 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-util\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-util\") pod \"af1176bd-e222-4009-9328-059825ea65a8\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " Nov 03 22:16:57.398786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.398683 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1176bd-e222-4009-9328-059825ea65a8-kube-api-access-gwrb2\" (UniqueName: \"kubernetes.io/projected/af1176bd-e222-4009-9328-059825ea65a8-kube-api-access-gwrb2\") pod \"af1176bd-e222-4009-9328-059825ea65a8\" (UID: \"af1176bd-e222-4009-9328-059825ea65a8\") " Nov 03 22:16:57.398972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.398952 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:57.399010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.398980 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fcsq\" (UniqueName: \"kubernetes.io/projected/06724f69-0c61-4139-a1fa-4628f6b8267b-kube-api-access-8fcsq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:57.399010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.398997 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06724f69-0c61-4139-a1fa-4628f6b8267b-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:57.399193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.399154 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-bundle" pod "af1176bd-e222-4009-9328-059825ea65a8" (UID: "af1176bd-e222-4009-9328-059825ea65a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:57.405653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.405620 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-util" pod "af1176bd-e222-4009-9328-059825ea65a8" (UID: "af1176bd-e222-4009-9328-059825ea65a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:57.410334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.410305 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1176bd-e222-4009-9328-059825ea65a8-kube-api-access-gwrb2" pod "af1176bd-e222-4009-9328-059825ea65a8" (UID: "af1176bd-e222-4009-9328-059825ea65a8"). InnerVolumeSpecName "kube-api-access-gwrb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:57.500410 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.500377 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwrb2\" (UniqueName: \"kubernetes.io/projected/af1176bd-e222-4009-9328-059825ea65a8-kube-api-access-gwrb2\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:57.500410 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.500406 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:57.500410 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:57.500415 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1176bd-e222-4009-9328-059825ea65a8-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:58.111008 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.110978 4625 generic.go:356] "Generic (PLEG): container finished" podID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerID="945d5509d0b3b733c2f7d9c2bda7589a2c0165b941aec8f95b542f8cf6609abd" exitCode=0 Nov 03 22:16:58.111454 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.111045 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" event={"ID":"3dd1c3de-b95b-4e82-9e9b-908e166430fa","Type":"ContainerDied","Data":"945d5509d0b3b733c2f7d9c2bda7589a2c0165b941aec8f95b542f8cf6609abd"} Nov 03 22:16:58.113014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.112996 4625 generic.go:356] "Generic (PLEG): container finished" podID="76f40792-fcce-43b0-95bc-5c984392888f" containerID="a459df8cf06c031b8981fdc49c5aa9277d6153ce8ee5c4a1724dee3c207eecb2" exitCode=0 Nov 03 22:16:58.113112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.113026 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" event={"ID":"76f40792-fcce-43b0-95bc-5c984392888f","Type":"ContainerDied","Data":"a459df8cf06c031b8981fdc49c5aa9277d6153ce8ee5c4a1724dee3c207eecb2"} Nov 03 22:16:58.114866 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.114850 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" Nov 03 22:16:58.114955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.114881 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpksxh" event={"ID":"06724f69-0c61-4139-a1fa-4628f6b8267b","Type":"ContainerDied","Data":"99fd08cd967b8d1f00a7a237dbbeee7e688565a832e4b9c85b1be331ec7b55ea"} Nov 03 22:16:58.114955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.114917 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99fd08cd967b8d1f00a7a237dbbeee7e688565a832e4b9c85b1be331ec7b55ea" Nov 03 22:16:58.117101 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.117078 4625 generic.go:356] "Generic (PLEG): container finished" podID="82ecdb0a-033a-4433-9491-7478d3494b88" containerID="dbe9edb30b1257184e71fbaeca31f76a786e38849effb1acf0dbe14f23fc0090" exitCode=0 Nov 03 22:16:58.117178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.117102 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" event={"ID":"82ecdb0a-033a-4433-9491-7478d3494b88","Type":"ContainerDied","Data":"dbe9edb30b1257184e71fbaeca31f76a786e38849effb1acf0dbe14f23fc0090"} Nov 03 22:16:58.118706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.118689 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:16:58.119353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.119333 4625 generic.go:356] "Generic (PLEG): container finished" podID="26884d2c-6314-4cf8-b455-43dedd245166" containerID="e03ba444bebf953e148fc20e5292ee435340e01276f5d8da16a08b778fbca5ba" exitCode=1 Nov 03 22:16:58.119420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.119406 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" event={"ID":"26884d2c-6314-4cf8-b455-43dedd245166","Type":"ContainerDied","Data":"e03ba444bebf953e148fc20e5292ee435340e01276f5d8da16a08b778fbca5ba"} Nov 03 22:16:58.121118 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.121101 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" event={"ID":"af1176bd-e222-4009-9328-059825ea65a8","Type":"ContainerDied","Data":"b9efdcb45df11d688c09f67557b24e7cab6dbece00c211d297c16295fb625699"} Nov 03 22:16:58.121192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.121126 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bh5xx" Nov 03 22:16:58.121192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.121132 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9efdcb45df11d688c09f67557b24e7cab6dbece00c211d297c16295fb625699" Nov 03 22:16:58.237908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.237883 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:58.307944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.307900 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-bundle\") pod \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " Nov 03 22:16:58.308109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.307968 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d37266-e999-4a6c-9451-c1dfec5d62b0-kube-api-access-b256g\" (UniqueName: \"kubernetes.io/projected/f6d37266-e999-4a6c-9451-c1dfec5d62b0-kube-api-access-b256g\") pod \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " Nov 03 22:16:58.308109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.308006 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-util\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-util\") pod \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\" (UID: \"f6d37266-e999-4a6c-9451-c1dfec5d62b0\") " Nov 03 22:16:58.308648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.308614 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-bundle" pod "f6d37266-e999-4a6c-9451-c1dfec5d62b0" (UID: "f6d37266-e999-4a6c-9451-c1dfec5d62b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:58.310056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.310029 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d37266-e999-4a6c-9451-c1dfec5d62b0-kube-api-access-b256g" pod "f6d37266-e999-4a6c-9451-c1dfec5d62b0" (UID: "f6d37266-e999-4a6c-9451-c1dfec5d62b0"). InnerVolumeSpecName "kube-api-access-b256g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:58.315894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.315868 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-util" pod "f6d37266-e999-4a6c-9451-c1dfec5d62b0" (UID: "f6d37266-e999-4a6c-9451-c1dfec5d62b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:58.408815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.408707 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:58.408815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.408759 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b256g\" (UniqueName: \"kubernetes.io/projected/f6d37266-e999-4a6c-9451-c1dfec5d62b0-kube-api-access-b256g\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:58.408815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:58.408773 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6d37266-e999-4a6c-9451-c1dfec5d62b0-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.127557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.127525 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" Nov 03 22:16:59.127988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.127572 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l9hpd" event={"ID":"f6d37266-e999-4a6c-9451-c1dfec5d62b0","Type":"ContainerDied","Data":"6dd556b11c855bf2e79768aa1ae3bb47026446d05a120a63feba49d341ea427c"} Nov 03 22:16:59.127988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.127618 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd556b11c855bf2e79768aa1ae3bb47026446d05a120a63feba49d341ea427c" Nov 03 22:16:59.292894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.292858 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:16:59.321032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.321003 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-util\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-util\") pod \"82ecdb0a-033a-4433-9491-7478d3494b88\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " Nov 03 22:16:59.321139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.321059 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ecdb0a-033a-4433-9491-7478d3494b88-kube-api-access-hwbjs\" (UniqueName: \"kubernetes.io/projected/82ecdb0a-033a-4433-9491-7478d3494b88-kube-api-access-hwbjs\") pod \"82ecdb0a-033a-4433-9491-7478d3494b88\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " Nov 03 22:16:59.321139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.321104 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-bundle\") pod \"82ecdb0a-033a-4433-9491-7478d3494b88\" (UID: \"82ecdb0a-033a-4433-9491-7478d3494b88\") " Nov 03 22:16:59.321499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.321479 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-bundle" pod "82ecdb0a-033a-4433-9491-7478d3494b88" (UID: "82ecdb0a-033a-4433-9491-7478d3494b88"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.323071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.323045 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ecdb0a-033a-4433-9491-7478d3494b88-kube-api-access-hwbjs" pod "82ecdb0a-033a-4433-9491-7478d3494b88" (UID: "82ecdb0a-033a-4433-9491-7478d3494b88"). InnerVolumeSpecName "kube-api-access-hwbjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:59.327419 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.327393 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-util" pod "82ecdb0a-033a-4433-9491-7478d3494b88" (UID: "82ecdb0a-033a-4433-9491-7478d3494b88"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.341300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.341272 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:16:59.342862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.342838 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:16:59.344827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.344806 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:16:59.347684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.347662 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:16:59.421697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421623 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-util\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-util\") pod \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " Nov 03 22:16:59.421697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421660 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-bundle\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-bundle\") pod \"26884d2c-6314-4cf8-b455-43dedd245166\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " Nov 03 22:16:59.421947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421698 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-bundle\") pod \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " Nov 03 22:16:59.421947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421750 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26884d2c-6314-4cf8-b455-43dedd245166-kube-api-access-t5rkf\" (UniqueName: \"kubernetes.io/projected/26884d2c-6314-4cf8-b455-43dedd245166-kube-api-access-t5rkf\") pod \"26884d2c-6314-4cf8-b455-43dedd245166\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " Nov 03 22:16:59.421947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421791 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd1c3de-b95b-4e82-9e9b-908e166430fa-kube-api-access-qv6pr\" (UniqueName: \"kubernetes.io/projected/3dd1c3de-b95b-4e82-9e9b-908e166430fa-kube-api-access-qv6pr\") pod \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\" (UID: \"3dd1c3de-b95b-4e82-9e9b-908e166430fa\") " Nov 03 22:16:59.421947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421854 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-bundle\") pod \"76f40792-fcce-43b0-95bc-5c984392888f\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " Nov 03 22:16:59.421947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421888 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f40792-fcce-43b0-95bc-5c984392888f-kube-api-access-7rjks\" (UniqueName: \"kubernetes.io/projected/76f40792-fcce-43b0-95bc-5c984392888f-kube-api-access-7rjks\") pod \"76f40792-fcce-43b0-95bc-5c984392888f\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " Nov 03 22:16:59.421947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421939 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-util\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-util\") pod \"76f40792-fcce-43b0-95bc-5c984392888f\" (UID: \"76f40792-fcce-43b0-95bc-5c984392888f\") " Nov 03 22:16:59.422258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.421984 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-util\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-util\") pod \"26884d2c-6314-4cf8-b455-43dedd245166\" (UID: \"26884d2c-6314-4cf8-b455-43dedd245166\") " Nov 03 22:16:59.422258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.422197 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-bundle" pod "26884d2c-6314-4cf8-b455-43dedd245166" (UID: "26884d2c-6314-4cf8-b455-43dedd245166"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.422258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.422211 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.422258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.422256 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbjs\" (UniqueName: \"kubernetes.io/projected/82ecdb0a-033a-4433-9491-7478d3494b88-kube-api-access-hwbjs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.422468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.422274 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb0a-033a-4433-9491-7478d3494b88-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.423634 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.423586 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-bundle" pod "3dd1c3de-b95b-4e82-9e9b-908e166430fa" (UID: "3dd1c3de-b95b-4e82-9e9b-908e166430fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.424621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.424377 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-bundle" pod "76f40792-fcce-43b0-95bc-5c984392888f" (UID: "76f40792-fcce-43b0-95bc-5c984392888f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.424845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.424638 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f40792-fcce-43b0-95bc-5c984392888f-kube-api-access-7rjks" pod "76f40792-fcce-43b0-95bc-5c984392888f" (UID: "76f40792-fcce-43b0-95bc-5c984392888f"). InnerVolumeSpecName "kube-api-access-7rjks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:59.425187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.425150 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26884d2c-6314-4cf8-b455-43dedd245166-kube-api-access-t5rkf" pod "26884d2c-6314-4cf8-b455-43dedd245166" (UID: "26884d2c-6314-4cf8-b455-43dedd245166"). InnerVolumeSpecName "kube-api-access-t5rkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:59.425264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.425240 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd1c3de-b95b-4e82-9e9b-908e166430fa-kube-api-access-qv6pr" pod "3dd1c3de-b95b-4e82-9e9b-908e166430fa" (UID: "3dd1c3de-b95b-4e82-9e9b-908e166430fa"). InnerVolumeSpecName "kube-api-access-qv6pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:59.430876 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.430853 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-util" pod "76f40792-fcce-43b0-95bc-5c984392888f" (UID: "76f40792-fcce-43b0-95bc-5c984392888f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.432102 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.432078 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-util" pod "26884d2c-6314-4cf8-b455-43dedd245166" (UID: "26884d2c-6314-4cf8-b455-43dedd245166"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.432577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.432558 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-util" pod "3dd1c3de-b95b-4e82-9e9b-908e166430fa" (UID: "3dd1c3de-b95b-4e82-9e9b-908e166430fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:16:59.522904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522873 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.522904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522899 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.522904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522908 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dd1c3de-b95b-4e82-9e9b-908e166430fa-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.523119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522916 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5rkf\" (UniqueName: \"kubernetes.io/projected/26884d2c-6314-4cf8-b455-43dedd245166-kube-api-access-t5rkf\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.523119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522925 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv6pr\" (UniqueName: \"kubernetes.io/projected/3dd1c3de-b95b-4e82-9e9b-908e166430fa-kube-api-access-qv6pr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.523119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522934 4625 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-bundle\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.523119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522942 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rjks\" (UniqueName: \"kubernetes.io/projected/76f40792-fcce-43b0-95bc-5c984392888f-kube-api-access-7rjks\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.523119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522949 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76f40792-fcce-43b0-95bc-5c984392888f-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:16:59.523119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:16:59.522957 4625 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26884d2c-6314-4cf8-b455-43dedd245166-util\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:17:00.133779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.133732 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" event={"ID":"3dd1c3de-b95b-4e82-9e9b-908e166430fa","Type":"ContainerDied","Data":"d844b4352bc791566994b6c27e5146b51e884bcde00e8571fb11e2243d1c9043"} Nov 03 22:17:00.134239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.133785 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwdr4" Nov 03 22:17:00.134239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.133801 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d844b4352bc791566994b6c27e5146b51e884bcde00e8571fb11e2243d1c9043" Nov 03 22:17:00.135653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.135633 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" Nov 03 22:17:00.135776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.135657 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl5jlt" event={"ID":"76f40792-fcce-43b0-95bc-5c984392888f","Type":"ContainerDied","Data":"c4eba0b12c9d4645a12064b13ff8c302e2dae301442de4585bb2d85903c8449a"} Nov 03 22:17:00.135776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.135697 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4eba0b12c9d4645a12064b13ff8c302e2dae301442de4585bb2d85903c8449a" Nov 03 22:17:00.137679 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.137659 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" Nov 03 22:17:00.137806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.137661 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d827f" event={"ID":"82ecdb0a-033a-4433-9491-7478d3494b88","Type":"ContainerDied","Data":"cd568d079143845f6d684ed44f3c85336946ce3019c131bf4e02a6e6b5143f95"} Nov 03 22:17:00.137806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.137767 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd568d079143845f6d684ed44f3c85336946ce3019c131bf4e02a6e6b5143f95" Nov 03 22:17:00.139076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.139056 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:17:00.139814 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.139795 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" event={"ID":"26884d2c-6314-4cf8-b455-43dedd245166","Type":"ContainerDied","Data":"ad8c5be86e905eef384250e214c561ef6ef5b2b6ead7be874cac5aad4b7832d5"} Nov 03 22:17:00.139885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.139832 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad8c5be86e905eef384250e214c561ef6ef5b2b6ead7be874cac5aad4b7832d5" Nov 03 22:17:00.139938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:00.139898 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx" Nov 03 22:17:02.183898 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:17:02.183865 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:17:04.177394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177362 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs"] Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177650 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f7470de4-2187-46d6-9e08-9fc604e23039" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177661 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7470de4-2187-46d6-9e08-9fc604e23039" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177672 4625 cpu_manager.go:414] "containerMap: removing container" podUID="af1176bd-e222-4009-9328-059825ea65a8" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177678 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1176bd-e222-4009-9328-059825ea65a8" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177685 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177691 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177698 4625 cpu_manager.go:414] "containerMap: removing container" podUID="82ecdb0a-033a-4433-9491-7478d3494b88" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177703 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ecdb0a-033a-4433-9491-7478d3494b88" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177727 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f7470de4-2187-46d6-9e08-9fc604e23039" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177737 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7470de4-2187-46d6-9e08-9fc604e23039" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177746 4625 cpu_manager.go:414] "containerMap: removing container" podUID="af1176bd-e222-4009-9328-059825ea65a8" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177753 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1176bd-e222-4009-9328-059825ea65a8" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177770 4625 cpu_manager.go:414] "containerMap: removing container" podUID="26884d2c-6314-4cf8-b455-43dedd245166" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177775 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26884d2c-6314-4cf8-b455-43dedd245166" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177782 4625 cpu_manager.go:414] "containerMap: removing container" podUID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177786 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177790 4625 cpu_manager.go:414] "containerMap: removing container" podUID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177795 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerName="util" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177800 4625 cpu_manager.go:414] "containerMap: removing container" podUID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177804 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177808 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177813 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177819 4625 cpu_manager.go:414] "containerMap: removing container" podUID="26884d2c-6314-4cf8-b455-43dedd245166" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177823 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26884d2c-6314-4cf8-b455-43dedd245166" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177830 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177835 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177841 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177846 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177852 4625 cpu_manager.go:414] "containerMap: removing container" podUID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177856 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177863 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f7470de4-2187-46d6-9e08-9fc604e23039" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177867 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7470de4-2187-46d6-9e08-9fc604e23039" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177872 4625 cpu_manager.go:414] "containerMap: removing container" podUID="76f40792-fcce-43b0-95bc-5c984392888f" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177877 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f40792-fcce-43b0-95bc-5c984392888f" containerName="pull" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177882 4625 cpu_manager.go:414] "containerMap: removing container" podUID="76f40792-fcce-43b0-95bc-5c984392888f" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177887 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f40792-fcce-43b0-95bc-5c984392888f" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177892 4625 cpu_manager.go:414] "containerMap: removing container" podUID="26884d2c-6314-4cf8-b455-43dedd245166" containerName="extract" Nov 03 22:17:04.177883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177896 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26884d2c-6314-4cf8-b455-43dedd245166" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177905 4625 cpu_manager.go:414] "containerMap: removing container" podUID="82ecdb0a-033a-4433-9491-7478d3494b88" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177910 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ecdb0a-033a-4433-9491-7478d3494b88" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177916 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177921 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177927 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177931 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177938 4625 cpu_manager.go:414] "containerMap: removing container" podUID="af1176bd-e222-4009-9328-059825ea65a8" containerName="pull" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177942 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1176bd-e222-4009-9328-059825ea65a8" containerName="pull" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177949 4625 cpu_manager.go:414] "containerMap: removing container" podUID="76f40792-fcce-43b0-95bc-5c984392888f" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177953 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f40792-fcce-43b0-95bc-5c984392888f" containerName="util" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177959 4625 cpu_manager.go:414] "containerMap: removing container" podUID="82ecdb0a-033a-4433-9491-7478d3494b88" containerName="pull" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177963 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ecdb0a-033a-4433-9491-7478d3494b88" containerName="pull" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177971 4625 cpu_manager.go:414] "containerMap: removing container" podUID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177975 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177981 4625 cpu_manager.go:414] "containerMap: removing container" podUID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.177984 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178030 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d63026-0e16-45cf-be3c-325b7a00cb98" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178037 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="06724f69-0c61-4139-a1fa-4628f6b8267b" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178042 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7470de4-2187-46d6-9e08-9fc604e23039" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178047 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1176bd-e222-4009-9328-059825ea65a8" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178052 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd1c3de-b95b-4e82-9e9b-908e166430fa" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178057 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ecdb0a-033a-4433-9491-7478d3494b88" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178064 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d37266-e999-4a6c-9451-c1dfec5d62b0" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178070 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f40792-fcce-43b0-95bc-5c984392888f" containerName="extract" Nov 03 22:17:04.178972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.178075 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="26884d2c-6314-4cf8-b455-43dedd245166" containerName="extract" Nov 03 22:17:04.180518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.180501 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.195181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.195158 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:17:04.206535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.206504 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 22:17:04.208240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.208213 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs"] Nov 03 22:17:04.262112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.262081 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/762872f1-9b93-4e60-9e98-d71c98f5541f-tmp\") pod \"jobset-operator-674cd59dc5-kc7zs\" (UID: \"762872f1-9b93-4e60-9e98-d71c98f5541f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.262283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.262146 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzh2j\" (UniqueName: \"kubernetes.io/projected/762872f1-9b93-4e60-9e98-d71c98f5541f-kube-api-access-zzh2j\") pod \"jobset-operator-674cd59dc5-kc7zs\" (UID: \"762872f1-9b93-4e60-9e98-d71c98f5541f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.287922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.287894 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-76f4f\"" Nov 03 22:17:04.363454 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.363420 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/762872f1-9b93-4e60-9e98-d71c98f5541f-tmp\") pod \"jobset-operator-674cd59dc5-kc7zs\" (UID: \"762872f1-9b93-4e60-9e98-d71c98f5541f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.363628 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.363503 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzh2j\" (UniqueName: \"kubernetes.io/projected/762872f1-9b93-4e60-9e98-d71c98f5541f-kube-api-access-zzh2j\") pod \"jobset-operator-674cd59dc5-kc7zs\" (UID: \"762872f1-9b93-4e60-9e98-d71c98f5541f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.363943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.363916 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/762872f1-9b93-4e60-9e98-d71c98f5541f-tmp\") pod \"jobset-operator-674cd59dc5-kc7zs\" (UID: \"762872f1-9b93-4e60-9e98-d71c98f5541f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.385562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.385531 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzh2j\" (UniqueName: \"kubernetes.io/projected/762872f1-9b93-4e60-9e98-d71c98f5541f-kube-api-access-zzh2j\") pod \"jobset-operator-674cd59dc5-kc7zs\" (UID: \"762872f1-9b93-4e60-9e98-d71c98f5541f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.503414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.503330 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" Nov 03 22:17:04.804087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:04.804016 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs"] Nov 03 22:17:04.807417 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:04.807385 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762872f1_9b93_4e60_9e98_d71c98f5541f.slice/crio-1fea08015290859fe753a9a9624f3794266d6eb3defe4567e5ee2e4cc3fd4cc8 WatchSource:0}: Error finding container 1fea08015290859fe753a9a9624f3794266d6eb3defe4567e5ee2e4cc3fd4cc8: Status 404 returned error can't find the container with id 1fea08015290859fe753a9a9624f3794266d6eb3defe4567e5ee2e4cc3fd4cc8 Nov 03 22:17:05.163464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:05.163428 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" event={"ID":"762872f1-9b93-4e60-9e98-d71c98f5541f","Type":"ContainerStarted","Data":"1fea08015290859fe753a9a9624f3794266d6eb3defe4567e5ee2e4cc3fd4cc8"} Nov 03 22:17:07.174353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:07.174317 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" event={"ID":"762872f1-9b93-4e60-9e98-d71c98f5541f","Type":"ContainerStarted","Data":"69947a18031c355449f0137ccfcf9be0b0d4c28c5fc677783e3b0e0bbbd4e043"} Nov 03 22:17:07.212870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:07.212821 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kc7zs" podStartSLOduration=1.60801486 podStartE2EDuration="3.212805107s" totalImagesPullingTime="1.604790247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:07.212666723 +0000 UTC m=+248.642002716" watchObservedRunningTime="2026-11-03 22:17:07.212805107 +0000 UTC m=+248.642141096" Nov 03 22:17:09.876912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.876874 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww"] Nov 03 22:17:09.879416 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.879399 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:09.909802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.909768 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:17:09.909802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.909783 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:17:09.909802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.909777 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:17:09.910050 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.909836 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wmvl7\"" Nov 03 22:17:09.910050 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.909849 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:17:09.910050 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.909923 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:17:09.928732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:09.928691 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww"] Nov 03 22:17:10.011471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.011405 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.011471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.011446 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.011471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.011467 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.011781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.011606 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmr8j\" (UniqueName: \"kubernetes.io/projected/d5eb006f-e69c-4aeb-9d87-c09654544d2d-kube-api-access-rmr8j\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.011781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.011639 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5eb006f-e69c-4aeb-9d87-c09654544d2d-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.011781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.011666 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5eb006f-e69c-4aeb-9d87-c09654544d2d-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.112660 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.112627 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.112881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.112672 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.112881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.112793 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.112881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.112836 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmr8j\" (UniqueName: \"kubernetes.io/projected/d5eb006f-e69c-4aeb-9d87-c09654544d2d-kube-api-access-rmr8j\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.113049 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.112954 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5eb006f-e69c-4aeb-9d87-c09654544d2d-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.113049 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.112997 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5eb006f-e69c-4aeb-9d87-c09654544d2d-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.113419 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.113393 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5eb006f-e69c-4aeb-9d87-c09654544d2d-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.115183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.115155 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5eb006f-e69c-4aeb-9d87-c09654544d2d-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.115370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.115340 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.115559 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.115542 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.115635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.115620 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5eb006f-e69c-4aeb-9d87-c09654544d2d-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.138038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.137971 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmr8j\" (UniqueName: \"kubernetes.io/projected/d5eb006f-e69c-4aeb-9d87-c09654544d2d-kube-api-access-rmr8j\") pod \"opendatahub-operator-controller-manager-65cf5586b-wl4ww\" (UID: \"d5eb006f-e69c-4aeb-9d87-c09654544d2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.191836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.191805 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:10.558191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:10.558110 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww"] Nov 03 22:17:10.561192 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:10.561163 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eb006f_e69c_4aeb_9d87_c09654544d2d.slice/crio-661d337ae6f7fe1fd6defb2591fb6878b54055a4e9c5af17f6d95011c5684816 WatchSource:0}: Error finding container 661d337ae6f7fe1fd6defb2591fb6878b54055a4e9c5af17f6d95011c5684816: Status 404 returned error can't find the container with id 661d337ae6f7fe1fd6defb2591fb6878b54055a4e9c5af17f6d95011c5684816 Nov 03 22:17:11.197040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:11.197002 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" event={"ID":"d5eb006f-e69c-4aeb-9d87-c09654544d2d","Type":"ContainerStarted","Data":"661d337ae6f7fe1fd6defb2591fb6878b54055a4e9c5af17f6d95011c5684816"} Nov 03 22:17:14.213248 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:14.213215 4625 generic.go:356] "Generic (PLEG): container finished" podID="d5eb006f-e69c-4aeb-9d87-c09654544d2d" containerID="8b590379f67683828600e76e366dc1d76bd17c1a5d51313b923d14fb35bdf512" exitCode=0 Nov 03 22:17:14.213652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:14.213310 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" event={"ID":"d5eb006f-e69c-4aeb-9d87-c09654544d2d","Type":"ContainerDied","Data":"8b590379f67683828600e76e366dc1d76bd17c1a5d51313b923d14fb35bdf512"} Nov 03 22:17:15.220303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:15.220258 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" event={"ID":"d5eb006f-e69c-4aeb-9d87-c09654544d2d","Type":"ContainerStarted","Data":"f0b3ffefd919c261b8701205a9ab3fbef68d9463f307da02a917d96b8a284287"} Nov 03 22:17:15.220751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:15.220355 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:15.307440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:15.307385 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" podStartSLOduration=2.5213368259999998 podStartE2EDuration="6.307371793s" totalImagesPullingTime="3.312408469s" totalInitContainerRuntime="473.626498ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:15.295137275 +0000 UTC m=+256.724473307" watchObservedRunningTime="2026-11-03 22:17:15.307371793 +0000 UTC m=+256.736707785" Nov 03 22:17:20.142790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.142753 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh"] Nov 03 22:17:20.149237 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.149221 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.158937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.158909 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh"] Nov 03 22:17:20.167812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.167786 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:17:20.167941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.167816 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:17:20.167941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.167841 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zc882\"" Nov 03 22:17:20.301550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.301517 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33e2a61-e1e0-4827-a750-5be4dfb76505-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5hdvh\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.301742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.301574 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlfjq\" (UniqueName: \"kubernetes.io/projected/c33e2a61-e1e0-4827-a750-5be4dfb76505-kube-api-access-hlfjq\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5hdvh\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.402996 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.402907 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33e2a61-e1e0-4827-a750-5be4dfb76505-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5hdvh\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.402996 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.402983 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlfjq\" (UniqueName: \"kubernetes.io/projected/c33e2a61-e1e0-4827-a750-5be4dfb76505-kube-api-access-hlfjq\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5hdvh\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.403459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.403438 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33e2a61-e1e0-4827-a750-5be4dfb76505-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5hdvh\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.425521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.425496 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlfjq\" (UniqueName: \"kubernetes.io/projected/c33e2a61-e1e0-4827-a750-5be4dfb76505-kube-api-access-hlfjq\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5hdvh\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.462525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.462490 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:20.796309 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:20.796223 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh"] Nov 03 22:17:20.799756 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:20.799731 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33e2a61_e1e0_4827_a750_5be4dfb76505.slice/crio-5be02c4a6a935da145d8cdea4254c6a81f312ed791d5078cb8b9d2dd1d57675c WatchSource:0}: Error finding container 5be02c4a6a935da145d8cdea4254c6a81f312ed791d5078cb8b9d2dd1d57675c: Status 404 returned error can't find the container with id 5be02c4a6a935da145d8cdea4254c6a81f312ed791d5078cb8b9d2dd1d57675c Nov 03 22:17:21.246624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:21.246591 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" event={"ID":"c33e2a61-e1e0-4827-a750-5be4dfb76505","Type":"ContainerStarted","Data":"5be02c4a6a935da145d8cdea4254c6a81f312ed791d5078cb8b9d2dd1d57675c"} Nov 03 22:17:24.313038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.313003 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5"] Nov 03 22:17:24.316787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.316760 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" Nov 03 22:17:24.331514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.331484 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 22:17:24.342125 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.342093 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kvd9g\"" Nov 03 22:17:24.346163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.346091 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5"] Nov 03 22:17:24.441392 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.441353 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pls8\" (UniqueName: \"kubernetes.io/projected/3917df96-a8b6-4304-8761-6d23a5f4ea2b-kube-api-access-7pls8\") pod \"dns-operator-controller-manager-8557cbdb84-jxtx5\" (UID: \"3917df96-a8b6-4304-8761-6d23a5f4ea2b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" Nov 03 22:17:24.542465 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.542424 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pls8\" (UniqueName: \"kubernetes.io/projected/3917df96-a8b6-4304-8761-6d23a5f4ea2b-kube-api-access-7pls8\") pod \"dns-operator-controller-manager-8557cbdb84-jxtx5\" (UID: \"3917df96-a8b6-4304-8761-6d23a5f4ea2b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" Nov 03 22:17:24.589982 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.589943 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pls8\" (UniqueName: \"kubernetes.io/projected/3917df96-a8b6-4304-8761-6d23a5f4ea2b-kube-api-access-7pls8\") pod \"dns-operator-controller-manager-8557cbdb84-jxtx5\" (UID: \"3917df96-a8b6-4304-8761-6d23a5f4ea2b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" Nov 03 22:17:24.640162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.640138 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" Nov 03 22:17:24.943442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:24.943361 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5"] Nov 03 22:17:24.948877 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:24.948851 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3917df96_a8b6_4304_8761_6d23a5f4ea2b.slice/crio-18188e9593c8e46d85a369838f68c963ce5474f9965524f5519f986977be4353 WatchSource:0}: Error finding container 18188e9593c8e46d85a369838f68c963ce5474f9965524f5519f986977be4353: Status 404 returned error can't find the container with id 18188e9593c8e46d85a369838f68c963ce5474f9965524f5519f986977be4353 Nov 03 22:17:25.271900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:25.271812 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" event={"ID":"c33e2a61-e1e0-4827-a750-5be4dfb76505","Type":"ContainerStarted","Data":"09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384"} Nov 03 22:17:25.272058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:25.271933 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:25.273209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:25.273187 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" event={"ID":"3917df96-a8b6-4304-8761-6d23a5f4ea2b","Type":"ContainerStarted","Data":"18188e9593c8e46d85a369838f68c963ce5474f9965524f5519f986977be4353"} Nov 03 22:17:25.313758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:25.313686 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" podStartSLOduration=1.483037839 podStartE2EDuration="5.313674384s" totalImagesPullingTime="3.830636545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:25.313170822 +0000 UTC m=+266.742506814" watchObservedRunningTime="2026-11-03 22:17:25.313674384 +0000 UTC m=+266.743010377" Nov 03 22:17:26.226778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:26.226747 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-wl4ww" Nov 03 22:17:27.721501 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.721469 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk"] Nov 03 22:17:27.725668 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.725649 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:27.741947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.741918 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wc98m\"" Nov 03 22:17:27.753475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.753452 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:17:27.753633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.753522 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 22:17:27.756145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.756122 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk"] Nov 03 22:17:27.873428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.873396 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s7ml\" (UniqueName: \"kubernetes.io/projected/95bcb5d0-cba2-4c8f-950c-9da5d5128903-kube-api-access-8s7ml\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk\" (UID: \"95bcb5d0-cba2-4c8f-950c-9da5d5128903\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:27.873611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.873442 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bcb5d0-cba2-4c8f-950c-9da5d5128903-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk\" (UID: \"95bcb5d0-cba2-4c8f-950c-9da5d5128903\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:27.974660 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.974572 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s7ml\" (UniqueName: \"kubernetes.io/projected/95bcb5d0-cba2-4c8f-950c-9da5d5128903-kube-api-access-8s7ml\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk\" (UID: \"95bcb5d0-cba2-4c8f-950c-9da5d5128903\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:27.974660 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.974622 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bcb5d0-cba2-4c8f-950c-9da5d5128903-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk\" (UID: \"95bcb5d0-cba2-4c8f-950c-9da5d5128903\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:27.975011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:27.974994 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bcb5d0-cba2-4c8f-950c-9da5d5128903-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk\" (UID: \"95bcb5d0-cba2-4c8f-950c-9da5d5128903\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:28.002017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.001988 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s7ml\" (UniqueName: \"kubernetes.io/projected/95bcb5d0-cba2-4c8f-950c-9da5d5128903-kube-api-access-8s7ml\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk\" (UID: \"95bcb5d0-cba2-4c8f-950c-9da5d5128903\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:28.037825 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.037795 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" Nov 03 22:17:28.289514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.289423 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" event={"ID":"3917df96-a8b6-4304-8761-6d23a5f4ea2b","Type":"ContainerStarted","Data":"c24c8d397b369b8201652e0f214c2eea765d2d474c28781a84f046faad8dbde4"} Nov 03 22:17:28.289675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.289550 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" Nov 03 22:17:28.330170 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.330116 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" podStartSLOduration=1.491794976 podStartE2EDuration="4.330100821s" totalImagesPullingTime="2.838305845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:28.328859477 +0000 UTC m=+269.758195470" watchObservedRunningTime="2026-11-03 22:17:28.330100821 +0000 UTC m=+269.759436813" Nov 03 22:17:28.371952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.371920 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk"] Nov 03 22:17:28.375483 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:28.375455 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bcb5d0_cba2_4c8f_950c_9da5d5128903.slice/crio-a78316864b83dc84911d6928f977fc89c332a163401e64ed775233fe1b6d0644 WatchSource:0}: Error finding container a78316864b83dc84911d6928f977fc89c332a163401e64ed775233fe1b6d0644: Status 404 returned error can't find the container with id a78316864b83dc84911d6928f977fc89c332a163401e64ed775233fe1b6d0644 Nov 03 22:17:28.584340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.584257 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx"] Nov 03 22:17:28.589220 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.589198 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:17:28.604879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.604856 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-77z5v\"" Nov 03 22:17:28.607247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.607222 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx"] Nov 03 22:17:28.680704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.680668 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7pb\" (UniqueName: \"kubernetes.io/projected/7107b00f-9a1b-4b01-a596-b450fa81b1f4-kube-api-access-2f7pb\") pod \"limitador-operator-controller-manager-5778bb4449-xn8cx\" (UID: \"7107b00f-9a1b-4b01-a596-b450fa81b1f4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:17:28.781954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.781892 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7pb\" (UniqueName: \"kubernetes.io/projected/7107b00f-9a1b-4b01-a596-b450fa81b1f4-kube-api-access-2f7pb\") pod \"limitador-operator-controller-manager-5778bb4449-xn8cx\" (UID: \"7107b00f-9a1b-4b01-a596-b450fa81b1f4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:17:28.807230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.807199 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7pb\" (UniqueName: \"kubernetes.io/projected/7107b00f-9a1b-4b01-a596-b450fa81b1f4-kube-api-access-2f7pb\") pod \"limitador-operator-controller-manager-5778bb4449-xn8cx\" (UID: \"7107b00f-9a1b-4b01-a596-b450fa81b1f4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:17:28.902104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:28.902076 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:17:29.205135 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:29.205072 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx"] Nov 03 22:17:29.208145 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:29.208108 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7107b00f_9a1b_4b01_a596_b450fa81b1f4.slice/crio-88da94a6b145867214310a4c7cfe0f482ed96b509c8545f0ca7c84b4504e59e0 WatchSource:0}: Error finding container 88da94a6b145867214310a4c7cfe0f482ed96b509c8545f0ca7c84b4504e59e0: Status 404 returned error can't find the container with id 88da94a6b145867214310a4c7cfe0f482ed96b509c8545f0ca7c84b4504e59e0 Nov 03 22:17:29.296616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:29.296580 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" event={"ID":"95bcb5d0-cba2-4c8f-950c-9da5d5128903","Type":"ContainerStarted","Data":"a78316864b83dc84911d6928f977fc89c332a163401e64ed775233fe1b6d0644"} Nov 03 22:17:29.298284 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:29.298245 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" event={"ID":"7107b00f-9a1b-4b01-a596-b450fa81b1f4","Type":"ContainerStarted","Data":"88da94a6b145867214310a4c7cfe0f482ed96b509c8545f0ca7c84b4504e59e0"} Nov 03 22:17:30.310784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:30.310679 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" event={"ID":"95bcb5d0-cba2-4c8f-950c-9da5d5128903","Type":"ContainerStarted","Data":"7f9c58bb25d8c5bb0b1be436ae55fe2603b2927fa802054a6d1ce5d3017c58cd"} Nov 03 22:17:31.317622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:31.317587 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" event={"ID":"7107b00f-9a1b-4b01-a596-b450fa81b1f4","Type":"ContainerStarted","Data":"ed622a5a7836e2cbc4d521d835725899c344b71e639ef32117e3f955f93f995b"} Nov 03 22:17:31.318100 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:31.317684 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:17:31.372334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:31.372274 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-w7tpk" podStartSLOduration=2.738540572 podStartE2EDuration="4.37225789s" totalImagesPullingTime="1.633717318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:30.390774408 +0000 UTC m=+271.820110398" watchObservedRunningTime="2026-11-03 22:17:31.37225789 +0000 UTC m=+272.801593883" Nov 03 22:17:31.372539 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:31.372520 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" podStartSLOduration=1.357379629 podStartE2EDuration="3.372515207s" totalImagesPullingTime="2.015135578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:31.369690168 +0000 UTC m=+272.799026161" watchObservedRunningTime="2026-11-03 22:17:31.372515207 +0000 UTC m=+272.801851245" Nov 03 22:17:32.186175 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:17:32.186137 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:17:33.704336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:33.704300 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj"] Nov 03 22:17:33.707785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:33.707766 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" Nov 03 22:17:33.737079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:33.737054 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-df2d6\"" Nov 03 22:17:33.739076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:33.739054 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj"] Nov 03 22:17:33.826036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:33.826000 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzlm\" (UniqueName: \"kubernetes.io/projected/7f061ce4-a7b7-4972-bd61-c812f53861a6-kube-api-access-hjzlm\") pod \"authorino-operator-7b7bb4bc77-cg9nj\" (UID: \"7f061ce4-a7b7-4972-bd61-c812f53861a6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" Nov 03 22:17:33.926478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:33.926443 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzlm\" (UniqueName: \"kubernetes.io/projected/7f061ce4-a7b7-4972-bd61-c812f53861a6-kube-api-access-hjzlm\") pod \"authorino-operator-7b7bb4bc77-cg9nj\" (UID: \"7f061ce4-a7b7-4972-bd61-c812f53861a6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" Nov 03 22:17:33.949054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:33.949021 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzlm\" (UniqueName: \"kubernetes.io/projected/7f061ce4-a7b7-4972-bd61-c812f53861a6-kube-api-access-hjzlm\") pod \"authorino-operator-7b7bb4bc77-cg9nj\" (UID: \"7f061ce4-a7b7-4972-bd61-c812f53861a6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" Nov 03 22:17:34.020800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:34.020683 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" Nov 03 22:17:34.338672 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:34.338579 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj"] Nov 03 22:17:34.341536 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:34.341507 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f061ce4_a7b7_4972_bd61_c812f53861a6.slice/crio-6cb4b26252f84884fea4a67de3ab706f7e17d2f5d7d5cd8e48b48774b8620e74 WatchSource:0}: Error finding container 6cb4b26252f84884fea4a67de3ab706f7e17d2f5d7d5cd8e48b48774b8620e74: Status 404 returned error can't find the container with id 6cb4b26252f84884fea4a67de3ab706f7e17d2f5d7d5cd8e48b48774b8620e74 Nov 03 22:17:35.340228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:35.340196 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" event={"ID":"7f061ce4-a7b7-4972-bd61-c812f53861a6","Type":"ContainerStarted","Data":"6cb4b26252f84884fea4a67de3ab706f7e17d2f5d7d5cd8e48b48774b8620e74"} Nov 03 22:17:36.280617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.280535 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:36.355171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.355110 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" event={"ID":"7f061ce4-a7b7-4972-bd61-c812f53861a6","Type":"ContainerStarted","Data":"13723cb5f4682b3a2439c7666689d980cb39f87b5c418831183c54f755d0e0fa"} Nov 03 22:17:36.355645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.355571 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" Nov 03 22:17:36.420623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.420561 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" podStartSLOduration=1.817099316 podStartE2EDuration="3.420544394s" totalImagesPullingTime="1.603445078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:36.418880073 +0000 UTC m=+277.848216067" watchObservedRunningTime="2026-11-03 22:17:36.420544394 +0000 UTC m=+277.849880387" Nov 03 22:17:36.755678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.755643 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7"] Nov 03 22:17:36.761738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.761689 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:36.785126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.785093 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7"] Nov 03 22:17:36.853121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.853085 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pms5d\" (UniqueName: \"kubernetes.io/projected/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-kube-api-access-pms5d\") pod \"kuadrant-operator-controller-manager-678d87766f-rfdx7\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:36.853283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.853137 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rfdx7\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:36.954216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.954176 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pms5d\" (UniqueName: \"kubernetes.io/projected/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-kube-api-access-pms5d\") pod \"kuadrant-operator-controller-manager-678d87766f-rfdx7\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:36.954385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.954231 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rfdx7\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:36.954615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.954598 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rfdx7\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:36.978114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:36.978080 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pms5d\" (UniqueName: \"kubernetes.io/projected/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-kube-api-access-pms5d\") pod \"kuadrant-operator-controller-manager-678d87766f-rfdx7\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:37.074521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:37.074439 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:37.406652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:37.406621 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7"] Nov 03 22:17:37.424498 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:37.424464 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48f2e0c_8877_44f4_8c75_948fcb5f03d0.slice/crio-c0aa6f2c357adb1b0b7023e8388f9e878ea33e908d0f0bdf8498298b64c4841f WatchSource:0}: Error finding container c0aa6f2c357adb1b0b7023e8388f9e878ea33e908d0f0bdf8498298b64c4841f: Status 404 returned error can't find the container with id c0aa6f2c357adb1b0b7023e8388f9e878ea33e908d0f0bdf8498298b64c4841f Nov 03 22:17:38.367393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:38.367353 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" event={"ID":"d48f2e0c-8877-44f4-8c75-948fcb5f03d0","Type":"ContainerStarted","Data":"a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3"} Nov 03 22:17:38.367572 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:38.367409 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" event={"ID":"d48f2e0c-8877-44f4-8c75-948fcb5f03d0","Type":"ContainerStarted","Data":"c0aa6f2c357adb1b0b7023e8388f9e878ea33e908d0f0bdf8498298b64c4841f"} Nov 03 22:17:38.367572 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:38.367559 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:38.434133 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:38.434081 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" podStartSLOduration=2.434066688 podStartE2EDuration="2.434066688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:38.421465843 +0000 UTC m=+279.850801837" watchObservedRunningTime="2026-11-03 22:17:38.434066688 +0000 UTC m=+279.863402680" Nov 03 22:17:39.301323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:39.301295 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxtx5" Nov 03 22:17:42.326541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:42.326504 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:17:43.262360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.262320 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw"] Nov 03 22:17:43.265625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.265602 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.281666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.281643 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qqvgz\"" Nov 03 22:17:43.298017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.297992 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw"] Nov 03 22:17:43.311504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311479 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d899d6c6-8a9a-4f11-8063-5ece280dad35-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.311625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311524 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.311625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311611 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.311707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311648 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.311793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311775 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.311851 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311808 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.311920 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311898 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.311959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311936 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.312006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.311967 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8r9\" (UniqueName: \"kubernetes.io/projected/d899d6c6-8a9a-4f11-8063-5ece280dad35-kube-api-access-nx8r9\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.412631 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412599 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d899d6c6-8a9a-4f11-8063-5ece280dad35-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412648 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412764 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412793 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412876 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412905 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412966 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.412995 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.413105 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8r9\" (UniqueName: \"kubernetes.io/projected/d899d6c6-8a9a-4f11-8063-5ece280dad35-kube-api-access-nx8r9\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.413225 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.413425 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.413451 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d899d6c6-8a9a-4f11-8063-5ece280dad35-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.413481 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.413534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.413498 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.415247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.415223 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.415349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.415251 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.436082 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.436053 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8r9\" (UniqueName: \"kubernetes.io/projected/d899d6c6-8a9a-4f11-8063-5ece280dad35-kube-api-access-nx8r9\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.436761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.436745 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d899d6c6-8a9a-4f11-8063-5ece280dad35-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9crbcbw\" (UID: \"d899d6c6-8a9a-4f11-8063-5ece280dad35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.581163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.581075 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:43.913114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.913081 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw"] Nov 03 22:17:43.915233 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:43.915212 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd899d6c6_8a9a_4f11_8063_5ece280dad35.slice/crio-75ba119f1920f28acc1a59c24fb3808307696a26b182c73a21c7cc2d781a91ad WatchSource:0}: Error finding container 75ba119f1920f28acc1a59c24fb3808307696a26b182c73a21c7cc2d781a91ad: Status 404 returned error can't find the container with id 75ba119f1920f28acc1a59c24fb3808307696a26b182c73a21c7cc2d781a91ad Nov 03 22:17:43.917396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.917364 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:17:43.917488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.917430 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:17:43.917488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:43.917458 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:17:44.406304 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:44.406267 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" event={"ID":"d899d6c6-8a9a-4f11-8063-5ece280dad35","Type":"ContainerStarted","Data":"ca48326c50647e48e7277e2c4c934e58e70a4d073306cb19e2669a9be45f5b67"} Nov 03 22:17:44.406487 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:44.406326 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" event={"ID":"d899d6c6-8a9a-4f11-8063-5ece280dad35","Type":"ContainerStarted","Data":"75ba119f1920f28acc1a59c24fb3808307696a26b182c73a21c7cc2d781a91ad"} Nov 03 22:17:44.452995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:44.452942 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" podStartSLOduration=1.452926094 podStartE2EDuration="1.452926094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:44.449990239 +0000 UTC m=+285.879326232" watchObservedRunningTime="2026-11-03 22:17:44.452926094 +0000 UTC m=+285.882262088" Nov 03 22:17:44.582079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:44.582042 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:44.587036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:44.587010 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:45.413600 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:45.413565 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:45.414776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:45.414754 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9crbcbw" Nov 03 22:17:47.339784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.339751 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd"] Nov 03 22:17:47.348434 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.348410 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.363145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.363112 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cg9nj" Nov 03 22:17:47.365046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.365021 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd"] Nov 03 22:17:47.373832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.373811 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:17:47.373832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.373831 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:17:47.396804 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.396779 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 22:17:47.433462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.433435 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-2fjpk\"" Nov 03 22:17:47.448939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.448912 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdc6c\" (UniqueName: \"kubernetes.io/projected/c437aa1e-22e8-4e04-97d7-e8fcd06de290-kube-api-access-mdc6c\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.449069 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.449052 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c437aa1e-22e8-4e04-97d7-e8fcd06de290-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.449122 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.449104 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c437aa1e-22e8-4e04-97d7-e8fcd06de290-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.449195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.449178 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c437aa1e-22e8-4e04-97d7-e8fcd06de290-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.550291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.550251 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c437aa1e-22e8-4e04-97d7-e8fcd06de290-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.550450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.550318 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdc6c\" (UniqueName: \"kubernetes.io/projected/c437aa1e-22e8-4e04-97d7-e8fcd06de290-kube-api-access-mdc6c\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.550450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.550362 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c437aa1e-22e8-4e04-97d7-e8fcd06de290-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.550450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.550382 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c437aa1e-22e8-4e04-97d7-e8fcd06de290-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.550644 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c437aa1e-22e8-4e04-97d7-e8fcd06de290-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.550700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.550678 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c437aa1e-22e8-4e04-97d7-e8fcd06de290-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.551198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.551178 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c437aa1e-22e8-4e04-97d7-e8fcd06de290-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.600390 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.600326 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdc6c\" (UniqueName: \"kubernetes.io/projected/c437aa1e-22e8-4e04-97d7-e8fcd06de290-kube-api-access-mdc6c\") pod \"kserve-module-controller-manager-7566cf5fc8-7q6bd\" (UID: \"c437aa1e-22e8-4e04-97d7-e8fcd06de290\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.661340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.661302 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:47.992303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:47.992212 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd"] Nov 03 22:17:47.995699 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:17:47.995666 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc437aa1e_22e8_4e04_97d7_e8fcd06de290.slice/crio-cf6bf95bf4d9967e17801fd9857d7f27ddd930161d1fcb2621c0d663ee7036a6 WatchSource:0}: Error finding container cf6bf95bf4d9967e17801fd9857d7f27ddd930161d1fcb2621c0d663ee7036a6: Status 404 returned error can't find the container with id cf6bf95bf4d9967e17801fd9857d7f27ddd930161d1fcb2621c0d663ee7036a6 Nov 03 22:17:48.431634 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:48.431598 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" event={"ID":"c437aa1e-22e8-4e04-97d7-e8fcd06de290","Type":"ContainerStarted","Data":"cf6bf95bf4d9967e17801fd9857d7f27ddd930161d1fcb2621c0d663ee7036a6"} Nov 03 22:17:49.376612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.376582 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 22:17:49.469438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.469404 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh"] Nov 03 22:17:49.469841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.469651 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" podUID="c33e2a61-e1e0-4827-a750-5be4dfb76505" containerName="manager" containerID="cri-o://09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384" gracePeriod=10 Nov 03 22:17:49.758332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.758301 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:49.772355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.772318 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33e2a61-e1e0-4827-a750-5be4dfb76505-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33e2a61-e1e0-4827-a750-5be4dfb76505-extensions-socket-volume\") pod \"c33e2a61-e1e0-4827-a750-5be4dfb76505\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " Nov 03 22:17:49.772540 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.772417 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33e2a61-e1e0-4827-a750-5be4dfb76505-kube-api-access-hlfjq\" (UniqueName: \"kubernetes.io/projected/c33e2a61-e1e0-4827-a750-5be4dfb76505-kube-api-access-hlfjq\") pod \"c33e2a61-e1e0-4827-a750-5be4dfb76505\" (UID: \"c33e2a61-e1e0-4827-a750-5be4dfb76505\") " Nov 03 22:17:49.773005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.772969 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33e2a61-e1e0-4827-a750-5be4dfb76505-extensions-socket-volume" pod "c33e2a61-e1e0-4827-a750-5be4dfb76505" (UID: "c33e2a61-e1e0-4827-a750-5be4dfb76505"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:17:49.775271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.775242 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33e2a61-e1e0-4827-a750-5be4dfb76505-kube-api-access-hlfjq" pod "c33e2a61-e1e0-4827-a750-5be4dfb76505" (UID: "c33e2a61-e1e0-4827-a750-5be4dfb76505"). InnerVolumeSpecName "kube-api-access-hlfjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:17:49.873538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.873502 4625 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33e2a61-e1e0-4827-a750-5be4dfb76505-extensions-socket-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:17:49.873538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:49.873537 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlfjq\" (UniqueName: \"kubernetes.io/projected/c33e2a61-e1e0-4827-a750-5be4dfb76505-kube-api-access-hlfjq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:17:50.446278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.446228 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" event={"ID":"c437aa1e-22e8-4e04-97d7-e8fcd06de290","Type":"ContainerStarted","Data":"a95ed2f59f196afd8a803a36fb8cc7c621b4916978027f56e281e88a1301fe5f"} Nov 03 22:17:50.446481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.446344 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:17:50.447941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.447913 4625 generic.go:356] "Generic (PLEG): container finished" podID="c33e2a61-e1e0-4827-a750-5be4dfb76505" containerID="09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384" exitCode=0 Nov 03 22:17:50.448083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.447957 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" Nov 03 22:17:50.448083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.447981 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" event={"ID":"c33e2a61-e1e0-4827-a750-5be4dfb76505","Type":"ContainerDied","Data":"09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384"} Nov 03 22:17:50.448083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.448025 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh" event={"ID":"c33e2a61-e1e0-4827-a750-5be4dfb76505","Type":"ContainerDied","Data":"5be02c4a6a935da145d8cdea4254c6a81f312ed791d5078cb8b9d2dd1d57675c"} Nov 03 22:17:50.448083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.448055 4625 scope.go:122] "RemoveContainer" containerID="09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384" Nov 03 22:17:50.458290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.458267 4625 scope.go:122] "RemoveContainer" containerID="09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384" Nov 03 22:17:50.458633 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:17:50.458580 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384\": container with ID starting with 09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384 not found: ID does not exist" containerID="09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384" Nov 03 22:17:50.458748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.458641 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384"} err="failed to get container status \"09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384\": rpc error: code = NotFound desc = could not find container \"09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384\": container with ID starting with 09434758e1a4a434a708c7787f1fe6c9087c3d5908a9d475bfa4efac2004a384 not found: ID does not exist" Nov 03 22:17:50.498579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.498531 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" podStartSLOduration=1.206449974 podStartE2EDuration="3.49851584s" totalImagesPullingTime="2.292065866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:50.498045433 +0000 UTC m=+291.927381425" watchObservedRunningTime="2026-11-03 22:17:50.49851584 +0000 UTC m=+291.927851831" Nov 03 22:17:50.545288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.545199 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh"] Nov 03 22:17:50.563427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.563393 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5hdvh"] Nov 03 22:17:50.726336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:50.726300 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33e2a61-e1e0-4827-a750-5be4dfb76505" path="/var/lib/kubelet/pods/c33e2a61-e1e0-4827-a750-5be4dfb76505/volumes" Nov 03 22:17:58.734409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:58.734381 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:17:58.734871 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:17:58.734732 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:18:02.184750 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:18:02.184701 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:18:11.220566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.220488 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pwj64"] Nov 03 22:18:11.220950 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.220864 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c33e2a61-e1e0-4827-a750-5be4dfb76505" containerName="manager" Nov 03 22:18:11.220950 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.220875 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33e2a61-e1e0-4827-a750-5be4dfb76505" containerName="manager" Nov 03 22:18:11.221021 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.220953 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33e2a61-e1e0-4827-a750-5be4dfb76505" containerName="manager" Nov 03 22:18:11.222951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.222934 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pwj64" Nov 03 22:18:11.240334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.240308 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pwj64"] Nov 03 22:18:11.249313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.249276 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzppl\"" Nov 03 22:18:11.372047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.372008 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8zs\" (UniqueName: \"kubernetes.io/projected/4599dd72-1917-449a-a7df-ec57882d493e-kube-api-access-hc8zs\") pod \"authorino-57f7f8659f-pwj64\" (UID: \"4599dd72-1917-449a-a7df-ec57882d493e\") " pod="kuadrant-system/authorino-57f7f8659f-pwj64" Nov 03 22:18:11.459641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.459592 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7q6bd" Nov 03 22:18:11.473344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.473277 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8zs\" (UniqueName: \"kubernetes.io/projected/4599dd72-1917-449a-a7df-ec57882d493e-kube-api-access-hc8zs\") pod \"authorino-57f7f8659f-pwj64\" (UID: \"4599dd72-1917-449a-a7df-ec57882d493e\") " pod="kuadrant-system/authorino-57f7f8659f-pwj64" Nov 03 22:18:11.495452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.495419 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8zs\" (UniqueName: \"kubernetes.io/projected/4599dd72-1917-449a-a7df-ec57882d493e-kube-api-access-hc8zs\") pod \"authorino-57f7f8659f-pwj64\" (UID: \"4599dd72-1917-449a-a7df-ec57882d493e\") " pod="kuadrant-system/authorino-57f7f8659f-pwj64" Nov 03 22:18:11.533751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.533701 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pwj64" Nov 03 22:18:11.837266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.837189 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pwj64"] Nov 03 22:18:11.840490 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:18:11.840452 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4599dd72_1917_449a_a7df_ec57882d493e.slice/crio-6011b5da3235c9c833dd1675e9fad1ad27a48eeb2986b497ce711a22b9595bee WatchSource:0}: Error finding container 6011b5da3235c9c833dd1675e9fad1ad27a48eeb2986b497ce711a22b9595bee: Status 404 returned error can't find the container with id 6011b5da3235c9c833dd1675e9fad1ad27a48eeb2986b497ce711a22b9595bee Nov 03 22:18:11.841850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:11.841829 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:18:12.568458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:12.568425 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pwj64" event={"ID":"4599dd72-1917-449a-a7df-ec57882d493e","Type":"ContainerStarted","Data":"6011b5da3235c9c833dd1675e9fad1ad27a48eeb2986b497ce711a22b9595bee"} Nov 03 22:18:14.583829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:14.583790 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pwj64" event={"ID":"4599dd72-1917-449a-a7df-ec57882d493e","Type":"ContainerStarted","Data":"01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be"} Nov 03 22:18:14.624860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:14.624810 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-pwj64" podStartSLOduration=1.749594794 podStartE2EDuration="3.624795906s" totalImagesPullingTime="1.875201112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:14.622671887 +0000 UTC m=+316.052007881" watchObservedRunningTime="2026-11-03 22:18:14.624795906 +0000 UTC m=+316.054131936" Nov 03 22:18:16.107689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:16.107654 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pwj64"] Nov 03 22:18:16.592994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:16.592933 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-pwj64" podUID="4599dd72-1917-449a-a7df-ec57882d493e" containerName="authorino" containerID="cri-o://01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be" gracePeriod=30 Nov 03 22:18:16.839622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:16.839598 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pwj64" Nov 03 22:18:16.922902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:16.922826 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4599dd72-1917-449a-a7df-ec57882d493e-kube-api-access-hc8zs\" (UniqueName: \"kubernetes.io/projected/4599dd72-1917-449a-a7df-ec57882d493e-kube-api-access-hc8zs\") pod \"4599dd72-1917-449a-a7df-ec57882d493e\" (UID: \"4599dd72-1917-449a-a7df-ec57882d493e\") " Nov 03 22:18:16.925027 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:16.924993 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4599dd72-1917-449a-a7df-ec57882d493e-kube-api-access-hc8zs" pod "4599dd72-1917-449a-a7df-ec57882d493e" (UID: "4599dd72-1917-449a-a7df-ec57882d493e"). InnerVolumeSpecName "kube-api-access-hc8zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:17.024053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.024023 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8zs\" (UniqueName: \"kubernetes.io/projected/4599dd72-1917-449a-a7df-ec57882d493e-kube-api-access-hc8zs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:18:17.599384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.599343 4625 generic.go:356] "Generic (PLEG): container finished" podID="4599dd72-1917-449a-a7df-ec57882d493e" containerID="01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be" exitCode=0 Nov 03 22:18:17.599812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.599391 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pwj64" Nov 03 22:18:17.599812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.599391 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pwj64" event={"ID":"4599dd72-1917-449a-a7df-ec57882d493e","Type":"ContainerDied","Data":"01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be"} Nov 03 22:18:17.599812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.599435 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pwj64" event={"ID":"4599dd72-1917-449a-a7df-ec57882d493e","Type":"ContainerDied","Data":"6011b5da3235c9c833dd1675e9fad1ad27a48eeb2986b497ce711a22b9595bee"} Nov 03 22:18:17.599812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.599451 4625 scope.go:122] "RemoveContainer" containerID="01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be" Nov 03 22:18:17.611531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.611509 4625 scope.go:122] "RemoveContainer" containerID="01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be" Nov 03 22:18:17.611980 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:18:17.611962 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be\": container with ID starting with 01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be not found: ID does not exist" containerID="01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be" Nov 03 22:18:17.612039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.611990 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be"} err="failed to get container status \"01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be\": rpc error: code = NotFound desc = could not find container \"01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be\": container with ID starting with 01a07fbaeba7c10e5ddb8c0cb222ad797d2274450b99819cdf1f23f4b50310be not found: ID does not exist" Nov 03 22:18:17.667596 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.667559 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pwj64"] Nov 03 22:18:17.678607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:17.678577 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pwj64"] Nov 03 22:18:18.726408 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:18.726374 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4599dd72-1917-449a-a7df-ec57882d493e" path="/var/lib/kubelet/pods/4599dd72-1917-449a-a7df-ec57882d493e/volumes" Nov 03 22:18:31.544791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.544751 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd"] Nov 03 22:18:31.545404 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.545120 4625 cpu_manager.go:414] "containerMap: removing container" podUID="4599dd72-1917-449a-a7df-ec57882d493e" containerName="authorino" Nov 03 22:18:31.545404 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.545132 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4599dd72-1917-449a-a7df-ec57882d493e" containerName="authorino" Nov 03 22:18:31.545404 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.545207 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="4599dd72-1917-449a-a7df-ec57882d493e" containerName="authorino" Nov 03 22:18:31.553493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.553468 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:31.562605 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.562577 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de54298a-44f5-4c18-8800-cf79f2bab9c6-data\") pod \"postgres-7cdbf6dc44-rgzjd\" (UID: \"de54298a-44f5-4c18-8800-cf79f2bab9c6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:31.562893 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.562853 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjrk\" (UniqueName: \"kubernetes.io/projected/de54298a-44f5-4c18-8800-cf79f2bab9c6-kube-api-access-pmjrk\") pod \"postgres-7cdbf6dc44-rgzjd\" (UID: \"de54298a-44f5-4c18-8800-cf79f2bab9c6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:31.566123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.566096 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd"] Nov 03 22:18:31.574355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.574330 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 22:18:31.574355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.574331 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:18:31.578558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.578537 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:18:31.578558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.578543 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gzvq6\"" Nov 03 22:18:31.664093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.664059 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de54298a-44f5-4c18-8800-cf79f2bab9c6-data\") pod \"postgres-7cdbf6dc44-rgzjd\" (UID: \"de54298a-44f5-4c18-8800-cf79f2bab9c6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:31.664093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.664096 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjrk\" (UniqueName: \"kubernetes.io/projected/de54298a-44f5-4c18-8800-cf79f2bab9c6-kube-api-access-pmjrk\") pod \"postgres-7cdbf6dc44-rgzjd\" (UID: \"de54298a-44f5-4c18-8800-cf79f2bab9c6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:31.664476 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.664455 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de54298a-44f5-4c18-8800-cf79f2bab9c6-data\") pod \"postgres-7cdbf6dc44-rgzjd\" (UID: \"de54298a-44f5-4c18-8800-cf79f2bab9c6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:31.686598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.686566 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjrk\" (UniqueName: \"kubernetes.io/projected/de54298a-44f5-4c18-8800-cf79f2bab9c6-kube-api-access-pmjrk\") pod \"postgres-7cdbf6dc44-rgzjd\" (UID: \"de54298a-44f5-4c18-8800-cf79f2bab9c6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:31.867350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:31.867313 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:32.166317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:32.166238 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd"] Nov 03 22:18:32.169949 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:18:32.169916 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde54298a_44f5_4c18_8800_cf79f2bab9c6.slice/crio-65af321909c548c716df31b843fa8da3961c9e62bb6dc46174b34b9a69a781bf WatchSource:0}: Error finding container 65af321909c548c716df31b843fa8da3961c9e62bb6dc46174b34b9a69a781bf: Status 404 returned error can't find the container with id 65af321909c548c716df31b843fa8da3961c9e62bb6dc46174b34b9a69a781bf Nov 03 22:18:32.187372 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:18:32.187339 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:18:32.678899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:32.678863 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" event={"ID":"de54298a-44f5-4c18-8800-cf79f2bab9c6","Type":"ContainerStarted","Data":"65af321909c548c716df31b843fa8da3961c9e62bb6dc46174b34b9a69a781bf"} Nov 03 22:18:37.710494 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:37.710454 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" event={"ID":"de54298a-44f5-4c18-8800-cf79f2bab9c6","Type":"ContainerStarted","Data":"bf5c9896e136b3c817063c45cd598de7ec1d8213f1952dc49d1cef2f1fb222ed"} Nov 03 22:18:37.710993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:37.710517 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:37.749599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:37.749485 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" podStartSLOduration=1.4348504499999999 podStartE2EDuration="6.749468989s" totalImagesPullingTime="5.314618539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:37.748260101 +0000 UTC m=+339.177596093" watchObservedRunningTime="2026-11-03 22:18:37.749468989 +0000 UTC m=+339.178804981" Nov 03 22:18:43.600111 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.600077 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv"] Nov 03 22:18:43.602734 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.602703 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.616502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.616474 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:18:43.616637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.616535 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:18:43.616637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.616533 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:18:43.616753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.616647 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h854s\"" Nov 03 22:18:43.638171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.638139 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv"] Nov 03 22:18:43.681207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.681175 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.681207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.681213 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.681412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.681231 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.681412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.681294 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.681477 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.681409 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.681477 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.681461 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.681541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.681484 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8r7k\" (UniqueName: \"kubernetes.io/projected/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-kube-api-access-z8r7k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.745074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.745045 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rgzjd" Nov 03 22:18:43.782398 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.782371 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.782558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.782405 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.782558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.782513 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.782655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.782574 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.782655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.782607 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8r7k\" (UniqueName: \"kubernetes.io/projected/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-kube-api-access-z8r7k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.782795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.782691 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.782795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.782734 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.783347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.783317 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.785407 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.785379 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.785518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.785439 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.785596 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.785575 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.785675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.785575 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.805147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.805115 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.805287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.805121 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8r7k\" (UniqueName: \"kubernetes.io/projected/6394ee67-e20e-4eb2-99a1-0c2aef09e7cd-kube-api-access-z8r7k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vg6xv\" (UID: \"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:43.914164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:43.914085 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:44.234036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:44.234011 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv"] Nov 03 22:18:44.236781 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:18:44.236752 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6394ee67_e20e_4eb2_99a1_0c2aef09e7cd.slice/crio-a7b5957966bd39f60585cacd68c596cfb4cd51541b2a0ad6eb8278f3cdbc7ab8 WatchSource:0}: Error finding container a7b5957966bd39f60585cacd68c596cfb4cd51541b2a0ad6eb8278f3cdbc7ab8: Status 404 returned error can't find the container with id a7b5957966bd39f60585cacd68c596cfb4cd51541b2a0ad6eb8278f3cdbc7ab8 Nov 03 22:18:44.747626 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:44.747590 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" event={"ID":"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd","Type":"ContainerStarted","Data":"a7b5957966bd39f60585cacd68c596cfb4cd51541b2a0ad6eb8278f3cdbc7ab8"} Nov 03 22:18:46.799531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:46.799495 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:18:46.799851 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:46.799581 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:18:47.765886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:47.765847 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" event={"ID":"6394ee67-e20e-4eb2-99a1-0c2aef09e7cd","Type":"ContainerStarted","Data":"a467a6ad95b3b359fe0ad5a3b9d15b9897503701ae31628332cf8537ba08b122"} Nov 03 22:18:47.766145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:47.766033 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:18:47.767785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:47.767752 4625 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vg6xv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:18:47.767918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:47.767800 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" podUID="6394ee67-e20e-4eb2-99a1-0c2aef09e7cd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:18:47.821907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:47.821854 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" podStartSLOduration=2.261875045 podStartE2EDuration="4.821838705s" totalImagesPullingTime="2.55996366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:47.819285055 +0000 UTC m=+349.248621039" watchObservedRunningTime="2026-11-03 22:18:47.821838705 +0000 UTC m=+349.251174689" Nov 03 22:18:48.771622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:18:48.771592 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vg6xv" Nov 03 22:19:02.184197 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:19:02.184155 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:19:32.185408 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:19:32.185364 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:19:37.022061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:37.022027 4625 generic.go:356] "Generic (PLEG): container finished" podID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerID="821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7" exitCode=0 Nov 03 22:19:37.022478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:37.022098 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" event={"ID":"d82444d7-3c98-4ed4-8ee9-3e227757dbed","Type":"ContainerDied","Data":"821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7"} Nov 03 22:19:37.022478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:37.022440 4625 scope.go:122] "RemoveContainer" containerID="821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7" Nov 03 22:19:37.023513 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:37.023487 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:19:37.023580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:37.023553 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:19:37.023629 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:37.023585 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:19:37.771181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:37.771145 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:19:38.030074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:38.029984 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" event={"ID":"d82444d7-3c98-4ed4-8ee9-3e227757dbed","Type":"ContainerStarted","Data":"aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811"} Nov 03 22:19:38.599967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:38.599925 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:19:38.601066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:38.601043 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:19:38.601130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:38.601094 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 22:19:39.599645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:39.599606 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:19:39.600158 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:39.599673 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 22:19:40.599589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:40.599552 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:19:40.599811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:40.599608 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 22:19:41.599493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:41.599447 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:19:41.599701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:41.599522 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 22:19:42.599235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:42.599191 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:19:42.599758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:42.599271 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 22:19:43.603997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:43.603965 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:19:43.604366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:43.604229 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:19:43.605001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:19:43.604986 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:20:02.185314 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:20:02.185279 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:20:32.185592 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:20:32.185554 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:20:54.087012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.086972 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b"] Nov 03 22:20:54.090162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.090139 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.118755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.118706 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 22:20:54.118755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.118744 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lfpms\"" Nov 03 22:20:54.121174 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.121148 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b"] Nov 03 22:20:54.135689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.135650 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.135831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.135756 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8e5e86-4926-41f1-ba99-b2da2fab4889-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.135831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.135814 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.135944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.135926 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.136005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.135983 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.136066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.136054 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zlb\" (UniqueName: \"kubernetes.io/projected/2d8e5e86-4926-41f1-ba99-b2da2fab4889-kube-api-access-v9zlb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.186806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.186770 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 22:20:54.186806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.186780 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 22:20:54.237067 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237031 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zlb\" (UniqueName: \"kubernetes.io/projected/2d8e5e86-4926-41f1-ba99-b2da2fab4889-kube-api-access-v9zlb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237080 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237102 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8e5e86-4926-41f1-ba99-b2da2fab4889-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237143 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237174 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237192 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237503 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237529 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.237739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.237689 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.239366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.239344 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8e5e86-4926-41f1-ba99-b2da2fab4889-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.239624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.239607 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8e5e86-4926-41f1-ba99-b2da2fab4889-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.259080 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.259050 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zlb\" (UniqueName: \"kubernetes.io/projected/2d8e5e86-4926-41f1-ba99-b2da2fab4889-kube-api-access-v9zlb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qt78b\" (UID: \"2d8e5e86-4926-41f1-ba99-b2da2fab4889\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.411396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.411352 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:54.719158 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:54.719081 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b"] Nov 03 22:20:54.722480 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:20:54.722449 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8e5e86_4926_41f1_ba99_b2da2fab4889.slice/crio-5b369de8e3d3ce79d1c39528a5ec01eaf67215ea788d8cc64da4285d0d047412 WatchSource:0}: Error finding container 5b369de8e3d3ce79d1c39528a5ec01eaf67215ea788d8cc64da4285d0d047412: Status 404 returned error can't find the container with id 5b369de8e3d3ce79d1c39528a5ec01eaf67215ea788d8cc64da4285d0d047412 Nov 03 22:20:55.456641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:55.456599 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" event={"ID":"2d8e5e86-4926-41f1-ba99-b2da2fab4889","Type":"ContainerStarted","Data":"5b369de8e3d3ce79d1c39528a5ec01eaf67215ea788d8cc64da4285d0d047412"} Nov 03 22:20:56.464785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:56.464747 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" event={"ID":"2d8e5e86-4926-41f1-ba99-b2da2fab4889","Type":"ContainerStarted","Data":"c3378a03c0c103477ddc0adb819a7edc2f03bd3bc6c2675b2c2704c41f1c6b93"} Nov 03 22:20:56.506753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:56.506686 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" podStartSLOduration=1.322513123 podStartE2EDuration="2.506673333s" totalImagesPullingTime="1.18416021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:56.504787918 +0000 UTC m=+477.934123912" watchObservedRunningTime="2026-11-03 22:20:56.506673333 +0000 UTC m=+477.936009325" Nov 03 22:20:57.411647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.411615 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:20:57.891514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.891480 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8"] Nov 03 22:20:57.895772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.895749 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:57.920561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.920534 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 22:20:57.924537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.924496 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8"] Nov 03 22:20:57.969855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.969823 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:57.970025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.969951 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f072ea9a-f752-461e-85e1-3297b846fde0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:57.970093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.970017 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:57.970093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.970057 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvpb\" (UniqueName: \"kubernetes.io/projected/f072ea9a-f752-461e-85e1-3297b846fde0-kube-api-access-wkvpb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:57.970195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.970127 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:57.970242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:57.970203 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071148 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071311 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f072ea9a-f752-461e-85e1-3297b846fde0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071370 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071395 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvpb\" (UniqueName: \"kubernetes.io/projected/f072ea9a-f752-461e-85e1-3297b846fde0-kube-api-access-wkvpb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071422 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071450 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071632 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071702 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.071985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.071960 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.074486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.074460 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f072ea9a-f752-461e-85e1-3297b846fde0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.074783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.074676 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f072ea9a-f752-461e-85e1-3297b846fde0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.095686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.095658 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvpb\" (UniqueName: \"kubernetes.io/projected/f072ea9a-f752-461e-85e1-3297b846fde0-kube-api-access-wkvpb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwbc8\" (UID: \"f072ea9a-f752-461e-85e1-3297b846fde0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.212147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.212034 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:20:58.516345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:58.516267 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8"] Nov 03 22:20:58.519769 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:20:58.519743 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf072ea9a_f752_461e_85e1_3297b846fde0.slice/crio-23282a8b15606587f1a54209b9f1044e677d9a776b63e5e433da369394d631a3 WatchSource:0}: Error finding container 23282a8b15606587f1a54209b9f1044e677d9a776b63e5e433da369394d631a3: Status 404 returned error can't find the container with id 23282a8b15606587f1a54209b9f1044e677d9a776b63e5e433da369394d631a3 Nov 03 22:20:59.482216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:59.482174 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" event={"ID":"f072ea9a-f752-461e-85e1-3297b846fde0","Type":"ContainerStarted","Data":"071cd50b8c312409956a16b050aa2911c8ca69ee9fcac2bb61e6bf2789e0ad7e"} Nov 03 22:20:59.482686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:59.482270 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" event={"ID":"f072ea9a-f752-461e-85e1-3297b846fde0","Type":"ContainerStarted","Data":"23282a8b15606587f1a54209b9f1044e677d9a776b63e5e433da369394d631a3"} Nov 03 22:20:59.524543 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:20:59.524496 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" podStartSLOduration=2.283187586 podStartE2EDuration="2.524478815s" totalImagesPullingTime="241.291229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:59.524073098 +0000 UTC m=+480.953409091" watchObservedRunningTime="2026-11-03 22:20:59.524478815 +0000 UTC m=+480.953814809" Nov 03 22:21:00.212938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:00.212882 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:21:01.903269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:01.903231 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7c5g6"] Nov 03 22:21:01.906976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:01.906953 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:01.925420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:01.925372 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7c5g6"] Nov 03 22:21:01.933798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:01.933777 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 22:21:02.014282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.014240 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-home\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.014282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.014282 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.014549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.014375 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.014549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.014412 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2zl\" (UniqueName: \"kubernetes.io/projected/6f8dc406-d4b0-407a-bd99-069aa88016cf-kube-api-access-jd2zl\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.014655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.014604 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8dc406-d4b0-407a-bd99-069aa88016cf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.014755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.014657 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.115535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.115500 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-home\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.115535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.115539 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.115793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.115560 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.115793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.115590 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2zl\" (UniqueName: \"kubernetes.io/projected/6f8dc406-d4b0-407a-bd99-069aa88016cf-kube-api-access-jd2zl\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.115793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.115654 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8dc406-d4b0-407a-bd99-069aa88016cf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.115793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.115676 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.116104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.116045 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-home\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.116104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.116056 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.116264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.116126 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.118070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.118045 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8dc406-d4b0-407a-bd99-069aa88016cf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.118224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.118205 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8dc406-d4b0-407a-bd99-069aa88016cf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.137913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.137882 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2zl\" (UniqueName: \"kubernetes.io/projected/6f8dc406-d4b0-407a-bd99-069aa88016cf-kube-api-access-jd2zl\") pod \"echo-simulator-kserve-64b6f8f96c-7c5g6\" (UID: \"6f8dc406-d4b0-407a-bd99-069aa88016cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.185630 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:21:02.185542 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:21:02.223221 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.223180 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:02.526821 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:02.526748 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7c5g6"] Nov 03 22:21:02.529797 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:21:02.529764 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8dc406_d4b0_407a_bd99_069aa88016cf.slice/crio-e7f9dd1a0dbcbb6048fd59152ddaf54b6d67c10029abe193b3fe20ed2e9d6bb6 WatchSource:0}: Error finding container e7f9dd1a0dbcbb6048fd59152ddaf54b6d67c10029abe193b3fe20ed2e9d6bb6: Status 404 returned error can't find the container with id e7f9dd1a0dbcbb6048fd59152ddaf54b6d67c10029abe193b3fe20ed2e9d6bb6 Nov 03 22:21:03.507054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:03.507014 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" event={"ID":"6f8dc406-d4b0-407a-bd99-069aa88016cf","Type":"ContainerStarted","Data":"e7f9dd1a0dbcbb6048fd59152ddaf54b6d67c10029abe193b3fe20ed2e9d6bb6"} Nov 03 22:21:04.411777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:04.411707 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:21:04.423952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:04.423925 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:21:04.514846 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:04.514749 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" event={"ID":"6f8dc406-d4b0-407a-bd99-069aa88016cf","Type":"ContainerStarted","Data":"ce6768536cd3d232f95e5eddcb8ebc48735f784c43520ed868abced6da3d0735"} Nov 03 22:21:04.527343 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:04.527312 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" Nov 03 22:21:04.556659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:04.556610 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" podStartSLOduration=1.907924366 podStartE2EDuration="3.556582532s" totalImagesPullingTime="1.648658166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:04.555327981 +0000 UTC m=+485.984663975" watchObservedRunningTime="2026-11-03 22:21:04.556582532 +0000 UTC m=+485.985918525" Nov 03 22:21:05.224071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:05.224041 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:08.212735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:08.212689 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:21:08.225413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:08.225388 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:21:08.548506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:08.548413 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" Nov 03 22:21:12.224253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:12.224170 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:12.236701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:12.236673 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:12.575424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:12.575341 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" Nov 03 22:21:13.244847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.244811 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f"] Nov 03 22:21:13.248381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.248364 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.285832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.285804 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f"] Nov 03 22:21:13.429520 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429482 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.429688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429529 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwtj\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-kube-api-access-hrwtj\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.429761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429678 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.429761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429744 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.429844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429774 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.429844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429802 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.429844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429825 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.429979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.429951 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.430029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.430007 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530470 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530509 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530537 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530570 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530585 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530604 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530636 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530702 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.530836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.530757 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwtj\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-kube-api-access-hrwtj\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.531105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.531032 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.531105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.531080 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.531274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.531254 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.531345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.531316 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.531432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.531411 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.533063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.533035 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.533177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.533128 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.553139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.553109 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.553274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.553242 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwtj\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-kube-api-access-hrwtj\") pod \"maas-default-gateway-openshift-default-7df854bf67-5w59f\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.564932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.564881 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:13.900161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.900138 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f"] Nov 03 22:21:13.902652 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:21:13.902617 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dfde9f_d8e4_4e06_928c_559ecdae76e9.slice/crio-5f0d9bc43671a80d705d9c171375ff8b8da4a787a045d2c2c31926405f7e0817 WatchSource:0}: Error finding container 5f0d9bc43671a80d705d9c171375ff8b8da4a787a045d2c2c31926405f7e0817: Status 404 returned error can't find the container with id 5f0d9bc43671a80d705d9c171375ff8b8da4a787a045d2c2c31926405f7e0817 Nov 03 22:21:13.905217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.905180 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:21:13.905310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.905258 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:21:13.905310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:13.905297 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:21:14.578172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:14.578132 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" event={"ID":"e1dfde9f-d8e4-4e06-928c-559ecdae76e9","Type":"ContainerStarted","Data":"4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd"} Nov 03 22:21:14.578577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:14.578191 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" event={"ID":"e1dfde9f-d8e4-4e06-928c-559ecdae76e9","Type":"ContainerStarted","Data":"5f0d9bc43671a80d705d9c171375ff8b8da4a787a045d2c2c31926405f7e0817"} Nov 03 22:21:14.665549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:14.665481 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podStartSLOduration=1.665461172 podStartE2EDuration="1.665461172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:14.650237636 +0000 UTC m=+496.079573630" watchObservedRunningTime="2026-11-03 22:21:14.665461172 +0000 UTC m=+496.094797167" Nov 03 22:21:15.565078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:15.565035 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:15.566541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:15.566517 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:21:15.566665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:15.566574 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 22:21:16.528980 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.528943 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-4h6tx"] Nov 03 22:21:16.543737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.540005 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:16.566899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.566862 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:21:16.567086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.566940 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 22:21:16.571445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.571393 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-4h6tx"] Nov 03 22:21:16.578373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.578348 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-metrics-certs\") pod \"ai-gateway-controller-5d46d8f988-4h6tx\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:16.578487 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.578401 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjxh\" (UniqueName: \"kubernetes.io/projected/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-kube-api-access-zmjxh\") pod \"ai-gateway-controller-5d46d8f988-4h6tx\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:16.579409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.579389 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 22:21:16.579499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.579486 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gzwbf\"" Nov 03 22:21:16.579597 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.579584 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 22:21:16.679461 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.679429 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjxh\" (UniqueName: \"kubernetes.io/projected/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-kube-api-access-zmjxh\") pod \"ai-gateway-controller-5d46d8f988-4h6tx\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:16.679637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.679564 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-metrics-certs\") pod \"ai-gateway-controller-5d46d8f988-4h6tx\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:16.682008 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.681987 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-metrics-certs\") pod \"ai-gateway-controller-5d46d8f988-4h6tx\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:16.701303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.701279 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjxh\" (UniqueName: \"kubernetes.io/projected/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-kube-api-access-zmjxh\") pod \"ai-gateway-controller-5d46d8f988-4h6tx\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:16.884162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:16.884121 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:17.212823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:17.212745 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-4h6tx"] Nov 03 22:21:17.215876 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:21:17.215848 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b7cd0b_6ad5_479c_9c71_08aaaec5932c.slice/crio-b23bcba8a8b86cf27d97114cc314e25ad87d8087496a055df2dad37a66484799 WatchSource:0}: Error finding container b23bcba8a8b86cf27d97114cc314e25ad87d8087496a055df2dad37a66484799: Status 404 returned error can't find the container with id b23bcba8a8b86cf27d97114cc314e25ad87d8087496a055df2dad37a66484799 Nov 03 22:21:17.565323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:17.565224 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:21:17.565323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:17.565277 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 22:21:17.598874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:17.598837 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" event={"ID":"02b7cd0b-6ad5-479c-9c71-08aaaec5932c","Type":"ContainerStarted","Data":"b23bcba8a8b86cf27d97114cc314e25ad87d8087496a055df2dad37a66484799"} Nov 03 22:21:18.568748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:18.566776 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:21:18.568748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:18.566834 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 22:21:19.566737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:19.565878 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:21:19.566737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:19.565941 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 22:21:20.565489 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:20.565452 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:21:20.565975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:20.565516 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 22:21:20.633391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:20.633304 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" event={"ID":"02b7cd0b-6ad5-479c-9c71-08aaaec5932c","Type":"ContainerStarted","Data":"5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e"} Nov 03 22:21:20.633549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:20.633403 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:20.672711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:20.672653 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" podStartSLOduration=1.5195287610000001 podStartE2EDuration="4.672638252s" totalImagesPullingTime="3.153109491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:20.671459816 +0000 UTC m=+502.100795812" watchObservedRunningTime="2026-11-03 22:21:20.672638252 +0000 UTC m=+502.101974244" Nov 03 22:21:22.263614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:22.263577 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:22.264138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:22.264053 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:22.265377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:22.265357 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:21:28.304392 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.304360 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58"] Nov 03 22:21:28.310052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.310028 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.330497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.330466 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58"] Nov 03 22:21:28.416216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416184 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416216 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416238 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416362 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmnp7\" (UniqueName: \"kubernetes.io/projected/ef90202f-42a4-45ca-93b0-deebd7306fec-kube-api-access-pmnp7\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416480 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416530 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416551 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef90202f-42a4-45ca-93b0-deebd7306fec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416573 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.416639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.416599 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518005 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmnp7\" (UniqueName: \"kubernetes.io/projected/ef90202f-42a4-45ca-93b0-deebd7306fec-kube-api-access-pmnp7\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518078 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518119 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518148 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef90202f-42a4-45ca-93b0-deebd7306fec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518177 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518211 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518269 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518291 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518314 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518660 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518580 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518709 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518818 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.518925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.518898 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.519215 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.519188 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef90202f-42a4-45ca-93b0-deebd7306fec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.520599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.520577 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.520795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.520777 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.540537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.540508 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef90202f-42a4-45ca-93b0-deebd7306fec-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.540860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.540836 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmnp7\" (UniqueName: \"kubernetes.io/projected/ef90202f-42a4-45ca-93b0-deebd7306fec-kube-api-access-pmnp7\") pod \"maas-default-gateway-openshift-default-7df854bf67-zqt58\" (UID: \"ef90202f-42a4-45ca-93b0-deebd7306fec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.628671 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.628638 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:28.951828 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.951800 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58"] Nov 03 22:21:28.954395 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:21:28.954361 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef90202f_42a4_45ca_93b0_deebd7306fec.slice/crio-5d65a933a0b715e95294c1944bc5e4f2a931a478680e5cb97b29fd9adc2e3975 WatchSource:0}: Error finding container 5d65a933a0b715e95294c1944bc5e4f2a931a478680e5cb97b29fd9adc2e3975: Status 404 returned error can't find the container with id 5d65a933a0b715e95294c1944bc5e4f2a931a478680e5cb97b29fd9adc2e3975 Nov 03 22:21:28.956767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.956731 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:21:28.956857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.956801 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:21:28.956857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:28.956831 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:21:29.693209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:29.693167 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" event={"ID":"ef90202f-42a4-45ca-93b0-deebd7306fec","Type":"ContainerStarted","Data":"c0cc1acebd0ea9b22786b50107edaff84a0423909d90757fe50de9216d98094c"} Nov 03 22:21:29.693613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:29.693224 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" event={"ID":"ef90202f-42a4-45ca-93b0-deebd7306fec","Type":"ContainerStarted","Data":"5d65a933a0b715e95294c1944bc5e4f2a931a478680e5cb97b29fd9adc2e3975"} Nov 03 22:21:30.629472 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:30.629431 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:30.630931 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:30.630906 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:30.631057 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:30.630949 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 22:21:31.629567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:31.629525 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:31.630012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:31.629602 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 22:21:31.658059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:31.658025 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:31.711746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:31.711685 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podStartSLOduration=3.71166999 podStartE2EDuration="3.71166999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:29.762087472 +0000 UTC m=+511.191423465" watchObservedRunningTime="2026-11-03 22:21:31.71166999 +0000 UTC m=+513.141005985" Nov 03 22:21:31.980493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:31.980408 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5"] Nov 03 22:21:31.986919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:31.986895 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.004198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.004174 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5"] Nov 03 22:21:32.075445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.075406 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqrz\" (UniqueName: \"kubernetes.io/projected/4471242b-9cd4-400f-bfc7-d1075e4c432d-kube-api-access-lvqrz\") pod \"ai-gateway-controller-6c9975f7bf-bm2l5\" (UID: \"4471242b-9cd4-400f-bfc7-d1075e4c432d\") " pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.075636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.075569 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4471242b-9cd4-400f-bfc7-d1075e4c432d-metrics-certs\") pod \"ai-gateway-controller-6c9975f7bf-bm2l5\" (UID: \"4471242b-9cd4-400f-bfc7-d1075e4c432d\") " pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.176767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.176726 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4471242b-9cd4-400f-bfc7-d1075e4c432d-metrics-certs\") pod \"ai-gateway-controller-6c9975f7bf-bm2l5\" (UID: \"4471242b-9cd4-400f-bfc7-d1075e4c432d\") " pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.176941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.176841 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqrz\" (UniqueName: \"kubernetes.io/projected/4471242b-9cd4-400f-bfc7-d1075e4c432d-kube-api-access-lvqrz\") pod \"ai-gateway-controller-6c9975f7bf-bm2l5\" (UID: \"4471242b-9cd4-400f-bfc7-d1075e4c432d\") " pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.179264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.179239 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4471242b-9cd4-400f-bfc7-d1075e4c432d-metrics-certs\") pod \"ai-gateway-controller-6c9975f7bf-bm2l5\" (UID: \"4471242b-9cd4-400f-bfc7-d1075e4c432d\") " pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.185631 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:21:32.185602 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:21:32.199518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.199491 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqrz\" (UniqueName: \"kubernetes.io/projected/4471242b-9cd4-400f-bfc7-d1075e4c432d-kube-api-access-lvqrz\") pod \"ai-gateway-controller-6c9975f7bf-bm2l5\" (UID: \"4471242b-9cd4-400f-bfc7-d1075e4c432d\") " pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.300776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.300682 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:32.623917 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.623893 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5"] Nov 03 22:21:32.626205 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:21:32.626178 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4471242b_9cd4_400f_bfc7_d1075e4c432d.slice/crio-616e6c234bfcc68c7a577ab5284257b87e10f7a09ead948ca9b796fca30d834a WatchSource:0}: Error finding container 616e6c234bfcc68c7a577ab5284257b87e10f7a09ead948ca9b796fca30d834a: Status 404 returned error can't find the container with id 616e6c234bfcc68c7a577ab5284257b87e10f7a09ead948ca9b796fca30d834a Nov 03 22:21:32.629661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.629636 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:32.629991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.629676 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 22:21:32.710772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:32.710738 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" event={"ID":"4471242b-9cd4-400f-bfc7-d1075e4c432d","Type":"ContainerStarted","Data":"616e6c234bfcc68c7a577ab5284257b87e10f7a09ead948ca9b796fca30d834a"} Nov 03 22:21:33.629692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:33.629658 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:33.630125 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:33.629710 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 22:21:33.718743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:33.718696 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" event={"ID":"4471242b-9cd4-400f-bfc7-d1075e4c432d","Type":"ContainerStarted","Data":"6938b156517e9b543c645f9f69ea170492692ff137545558ad1431eaca44dd34"} Nov 03 22:21:33.718922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:33.718866 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:33.759489 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:33.759425 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" podStartSLOduration=2.387005186 podStartE2EDuration="2.759406989s" totalImagesPullingTime="372.401803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:33.759117829 +0000 UTC m=+515.188453823" watchObservedRunningTime="2026-11-03 22:21:33.759406989 +0000 UTC m=+515.188742983" Nov 03 22:21:34.629466 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:34.629432 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:34.629634 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:34.629488 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 22:21:35.629305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:35.629271 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:21:35.629895 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:35.629331 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 22:21:37.152335 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:37.152295 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:37.152850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:37.152650 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:37.153555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:37.153532 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" Nov 03 22:21:44.728142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:44.728116 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6c9975f7bf-bm2l5" Nov 03 22:21:44.796402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:44.796352 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-4h6tx"] Nov 03 22:21:44.796912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:44.796872 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" podUID="02b7cd0b-6ad5-479c-9c71-08aaaec5932c" containerName="manager" containerID="cri-o://5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e" gracePeriod=30 Nov 03 22:21:45.043456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.043433 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:45.110072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.110038 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-kube-api-access-zmjxh\" (UniqueName: \"kubernetes.io/projected/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-kube-api-access-zmjxh\") pod \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " Nov 03 22:21:45.110256 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.110188 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-metrics-certs\") pod \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\" (UID: \"02b7cd0b-6ad5-479c-9c71-08aaaec5932c\") " Nov 03 22:21:45.112222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.112188 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-kube-api-access-zmjxh" pod "02b7cd0b-6ad5-479c-9c71-08aaaec5932c" (UID: "02b7cd0b-6ad5-479c-9c71-08aaaec5932c"). InnerVolumeSpecName "kube-api-access-zmjxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:45.112349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.112243 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-metrics-certs" pod "02b7cd0b-6ad5-479c-9c71-08aaaec5932c" (UID: "02b7cd0b-6ad5-479c-9c71-08aaaec5932c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:21:45.211084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.211051 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:21:45.211084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.211080 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjxh\" (UniqueName: \"kubernetes.io/projected/02b7cd0b-6ad5-479c-9c71-08aaaec5932c-kube-api-access-zmjxh\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:21:45.786174 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.786135 4625 generic.go:356] "Generic (PLEG): container finished" podID="02b7cd0b-6ad5-479c-9c71-08aaaec5932c" containerID="5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e" exitCode=0 Nov 03 22:21:45.786641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.786185 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" Nov 03 22:21:45.786641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.786217 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" event={"ID":"02b7cd0b-6ad5-479c-9c71-08aaaec5932c","Type":"ContainerDied","Data":"5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e"} Nov 03 22:21:45.786641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.786271 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-4h6tx" event={"ID":"02b7cd0b-6ad5-479c-9c71-08aaaec5932c","Type":"ContainerDied","Data":"b23bcba8a8b86cf27d97114cc314e25ad87d8087496a055df2dad37a66484799"} Nov 03 22:21:45.786641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.786292 4625 scope.go:122] "RemoveContainer" containerID="5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e" Nov 03 22:21:45.796201 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.796183 4625 scope.go:122] "RemoveContainer" containerID="5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e" Nov 03 22:21:45.796469 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:21:45.796448 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e\": container with ID starting with 5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e not found: ID does not exist" containerID="5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e" Nov 03 22:21:45.796549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.796481 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e"} err="failed to get container status \"5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e\": rpc error: code = NotFound desc = could not find container \"5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e\": container with ID starting with 5daf3ebe4cb4b7dea06a69915f02bc17c2ad8d95945f81170576da3f4fb66d4e not found: ID does not exist" Nov 03 22:21:45.834872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.834846 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-4h6tx"] Nov 03 22:21:45.845981 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:45.845955 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-4h6tx"] Nov 03 22:21:46.725903 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:21:46.725870 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b7cd0b-6ad5-479c-9c71-08aaaec5932c" path="/var/lib/kubelet/pods/02b7cd0b-6ad5-479c-9c71-08aaaec5932c/volumes" Nov 03 22:22:02.185541 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:22:02.185504 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:22:12.059858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.059823 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw"] Nov 03 22:22:12.060468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.060362 4625 cpu_manager.go:414] "containerMap: removing container" podUID="02b7cd0b-6ad5-479c-9c71-08aaaec5932c" containerName="manager" Nov 03 22:22:12.060468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.060375 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b7cd0b-6ad5-479c-9c71-08aaaec5932c" containerName="manager" Nov 03 22:22:12.060468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.060463 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b7cd0b-6ad5-479c-9c71-08aaaec5932c" containerName="manager" Nov 03 22:22:12.065204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.065178 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084236 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084210 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084256 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084351 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084378 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6725a80-3ec7-436a-a031-240f2c83e623-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084400 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtxq\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-kube-api-access-wbtxq\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084473 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084520 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6725a80-3ec7-436a-a031-240f2c83e623-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084560 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.084625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.084594 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.090796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.090770 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-50d738-openshift-default-dockercfg-jhlmb\"" Nov 03 22:22:12.104526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.104496 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw"] Nov 03 22:22:12.185949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.185919 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.185999 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186044 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186082 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186112 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6725a80-3ec7-436a-a031-240f2c83e623-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186141 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtxq\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-kube-api-access-wbtxq\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186173 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186523 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186475 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186576 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6725a80-3ec7-436a-a031-240f2c83e623-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186677 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186645 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.186934 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.186914 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.187311 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.187289 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.187480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.187376 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.187602 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.187520 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6725a80-3ec7-436a-a031-240f2c83e623-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.189838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.189817 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6725a80-3ec7-436a-a031-240f2c83e623-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.193064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.193035 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2"] Nov 03 22:22:12.196537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.196519 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.210321 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.210298 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.210435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.210323 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtxq\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-kube-api-access-wbtxq\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.212142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.212120 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.216960 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.216935 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-553753-openshift-default-dockercfg-vgskd\"" Nov 03 22:22:12.221813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.221789 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2"] Nov 03 22:22:12.287670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.287638 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.287864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.287680 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.287864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.287705 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.287949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.287865 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.287949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.287896 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.287949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.287914 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.287949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.287930 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp45x\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-kube-api-access-wp45x\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.288086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.288003 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.288086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.288028 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.378917 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.378884 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:12.390059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.390034 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.390172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.390073 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.390172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.390101 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.390172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.390145 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp45x\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-kube-api-access-wp45x\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.390481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.390461 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.391427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.390473 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.391583 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.390909 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.391694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.391619 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.391794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.391756 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.391853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.391803 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.391853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.391839 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.392083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.392057 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.392264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.392239 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.392449 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.392419 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.392965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.392947 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.395888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.395869 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.415501 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.415235 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp45x\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-kube-api-access-wp45x\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.415501 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.415444 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-j5dc2\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.539938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.539899 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:12.632076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.631967 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9"] Nov 03 22:22:12.637571 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.637547 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.659796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.659764 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9"] Nov 03 22:22:12.675171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.675148 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5187e1-openshift-default-dockercfg-6g9br\"" Nov 03 22:22:12.695091 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695053 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695244 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695205 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af5050e-5e51-4d60-a172-64c2d154a063-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695246 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695263 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695294 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af5050e-5e51-4d60-a172-64c2d154a063-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695464 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695510 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695604 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxn4\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-kube-api-access-jpxn4\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.695676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.695646 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.715946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.715914 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw"] Nov 03 22:22:12.719014 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:22:12.718987 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6725a80_3ec7_436a_a031_240f2c83e623.slice/crio-e4786107802de5137398e851fc83db2c6a81ae80e7d97b3c5f0e2ec76c7c9a66 WatchSource:0}: Error finding container e4786107802de5137398e851fc83db2c6a81ae80e7d97b3c5f0e2ec76c7c9a66: Status 404 returned error can't find the container with id e4786107802de5137398e851fc83db2c6a81ae80e7d97b3c5f0e2ec76c7c9a66 Nov 03 22:22:12.721282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.721253 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:12.721359 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.721317 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:12.721359 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.721346 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:12.796636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.796610 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxn4\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-kube-api-access-jpxn4\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.796772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.796654 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.796772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.796679 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.796912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.796856 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af5050e-5e51-4d60-a172-64c2d154a063-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.796968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.796908 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.797030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.796964 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.797030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.797014 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af5050e-5e51-4d60-a172-64c2d154a063-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.797139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.797075 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.797139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.797108 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.797374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.797350 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.797470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.797447 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.798011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.797743 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.798117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.797938 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.798235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.798212 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af5050e-5e51-4d60-a172-64c2d154a063-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.799566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.799546 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.799760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.799737 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af5050e-5e51-4d60-a172-64c2d154a063-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.820078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.820051 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.820623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.820293 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxn4\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-kube-api-access-jpxn4\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:12.860561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.860522 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2"] Nov 03 22:22:12.862895 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:22:12.862865 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf225f6da_aad0_43cb_9fe8_6fab90bbf69b.slice/crio-6a7f7940d1c7985dcb215b32e72a2eeeda9c1b46da48edad4dee0d7e7ed038a6 WatchSource:0}: Error finding container 6a7f7940d1c7985dcb215b32e72a2eeeda9c1b46da48edad4dee0d7e7ed038a6: Status 404 returned error can't find the container with id 6a7f7940d1c7985dcb215b32e72a2eeeda9c1b46da48edad4dee0d7e7ed038a6 Nov 03 22:22:12.865951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.865911 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:12.866064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.865998 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:12.866064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.866044 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:12.908122 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.908093 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t"] Nov 03 22:22:12.912549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.912520 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:12.939017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.938987 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-88a4d9-openshift-default-dockercfg-ph54l\"" Nov 03 22:22:12.942166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.940602 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t"] Nov 03 22:22:12.947599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.947568 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" event={"ID":"e6725a80-3ec7-436a-a031-240f2c83e623","Type":"ContainerStarted","Data":"2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80"} Nov 03 22:22:12.947775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.947628 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" event={"ID":"e6725a80-3ec7-436a-a031-240f2c83e623","Type":"ContainerStarted","Data":"e4786107802de5137398e851fc83db2c6a81ae80e7d97b3c5f0e2ec76c7c9a66"} Nov 03 22:22:12.960667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.960634 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" event={"ID":"f225f6da-aad0-43cb-9fe8-6fab90bbf69b","Type":"ContainerStarted","Data":"6a7f7940d1c7985dcb215b32e72a2eeeda9c1b46da48edad4dee0d7e7ed038a6"} Nov 03 22:22:12.971522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.971479 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999403 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999466 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999498 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999548 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999639 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29q4\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-kube-api-access-q29q4\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999683 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999820 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999879 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.001850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:12.999992 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.037043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.036982 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podStartSLOduration=1.036960757 podStartE2EDuration="1.036960757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:13.035598553 +0000 UTC m=+554.464934546" watchObservedRunningTime="2026-11-03 22:22:13.036960757 +0000 UTC m=+554.466296752" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102262 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102317 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102361 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102433 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29q4\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-kube-api-access-q29q4\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102466 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102525 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102571 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102636 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.102707 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.103174 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.103869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.103438 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.126101 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.125429 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.126265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.126105 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.127551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.126953 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.127551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.127505 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.143646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.143533 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.166875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.166242 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.178084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.178029 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29q4\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-kube-api-access-q29q4\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.274760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.274696 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:13.329904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.329869 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9"] Nov 03 22:22:13.387258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.384598 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:13.388746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.388668 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:13.388866 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.388784 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:13.388866 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.388829 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:13.394494 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.394473 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:13.593594 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.591337 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t"] Nov 03 22:22:13.599156 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:22:13.599121 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb7bc4c_db84_4ea3_b6dd_75e2d983cd43.slice/crio-fac83083e0858d3d79c2f0e3a144047d206444bff1195b93e708c0f84fc3dd4d WatchSource:0}: Error finding container fac83083e0858d3d79c2f0e3a144047d206444bff1195b93e708c0f84fc3dd4d: Status 404 returned error can't find the container with id fac83083e0858d3d79c2f0e3a144047d206444bff1195b93e708c0f84fc3dd4d Nov 03 22:22:13.601791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.601751 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:13.601890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.601846 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:13.601944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.601900 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:13.969530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.969486 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" event={"ID":"2af5050e-5e51-4d60-a172-64c2d154a063","Type":"ContainerStarted","Data":"659a948bb9b9529f05c14af772ecb97f1f721a452308f4463928d83ea49b7abe"} Nov 03 22:22:13.969753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.969559 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" event={"ID":"2af5050e-5e51-4d60-a172-64c2d154a063","Type":"ContainerStarted","Data":"39858a830aff83abb37bb59d82a452096af646ad34563c593d4295d14ff41958"} Nov 03 22:22:13.971332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.971305 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" event={"ID":"f225f6da-aad0-43cb-9fe8-6fab90bbf69b","Type":"ContainerStarted","Data":"252d83aac6558a86207e4b9d8a14ba113749662c3957dd83dbcc913a59e02bcd"} Nov 03 22:22:13.971770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.971750 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:13.972974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.972948 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" event={"ID":"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43","Type":"ContainerStarted","Data":"9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0"} Nov 03 22:22:13.973075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.972993 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" event={"ID":"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43","Type":"ContainerStarted","Data":"fac83083e0858d3d79c2f0e3a144047d206444bff1195b93e708c0f84fc3dd4d"} Nov 03 22:22:13.973229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.973208 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:13.974090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:13.974075 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:14.018612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.018559 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" podStartSLOduration=2.018544948 podStartE2EDuration="2.018544948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:14.016049988 +0000 UTC m=+555.445385980" watchObservedRunningTime="2026-11-03 22:22:14.018544948 +0000 UTC m=+555.447880941" Nov 03 22:22:14.059110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.059039 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podStartSLOduration=2.059022769 podStartE2EDuration="2.059022769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:14.058749927 +0000 UTC m=+555.488085920" watchObservedRunningTime="2026-11-03 22:22:14.059022769 +0000 UTC m=+555.488358764" Nov 03 22:22:14.102375 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.102324 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podStartSLOduration=2.102310235 podStartE2EDuration="2.102310235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:14.10099826 +0000 UTC m=+555.530334254" watchObservedRunningTime="2026-11-03 22:22:14.102310235 +0000 UTC m=+555.531646228" Nov 03 22:22:14.295794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.289964 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:14.329388 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.326704 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:14.540852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.540822 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:14.546222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.546150 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:14.979288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.979260 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:14.981321 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.981295 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:14.981461 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.981347 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:14.981461 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.981375 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:14.982371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.982351 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:14.982452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.982402 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:14.982452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:14.982431 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:20.480186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.480148 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5"] Nov 03 22:22:20.482894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.482876 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:20.507018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.506988 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5"] Nov 03 22:22:20.514750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.514707 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e19850\"" Nov 03 22:22:20.556852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.556447 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e19850-dockercfg-487lr\"" Nov 03 22:22:20.606144 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.606115 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98jl\" (UniqueName: \"kubernetes.io/projected/b4d2e243-3b00-4807-8324-e14cb06f0d20-kube-api-access-s98jl\") pod \"payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:20.606313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.606274 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2e243-3b00-4807-8324-e14cb06f0d20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:20.707285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.707247 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2e243-3b00-4807-8324-e14cb06f0d20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:20.707462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.707340 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98jl\" (UniqueName: \"kubernetes.io/projected/b4d2e243-3b00-4807-8324-e14cb06f0d20-kube-api-access-s98jl\") pod \"payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:20.708087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.708058 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2e243-3b00-4807-8324-e14cb06f0d20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:20.729625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.729574 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98jl\" (UniqueName: \"kubernetes.io/projected/b4d2e243-3b00-4807-8324-e14cb06f0d20-kube-api-access-s98jl\") pod \"payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:20.802652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:20.798308 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:21.152165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.152135 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5"] Nov 03 22:22:21.155906 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:22:21.155875 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d2e243_3b00_4807_8324_e14cb06f0d20.slice/crio-1b164e64e1d174dee9530f0a8d2b7f3926c62034d58ddab44c99c890f080f88c WatchSource:0}: Error finding container 1b164e64e1d174dee9530f0a8d2b7f3926c62034d58ddab44c99c890f080f88c: Status 404 returned error can't find the container with id 1b164e64e1d174dee9530f0a8d2b7f3926c62034d58ddab44c99c890f080f88c Nov 03 22:22:21.451551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.451429 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq"] Nov 03 22:22:21.462187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.459830 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:21.478225 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.478195 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq"] Nov 03 22:22:21.486010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.485039 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-553753-dockercfg-tm7gg\"" Nov 03 22:22:21.486010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.485158 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-553753\"" Nov 03 22:22:21.544056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.543876 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvgz\" (UniqueName: \"kubernetes.io/projected/701abc1a-231d-42a0-a997-ba3e21759e37-kube-api-access-ffvgz\") pod \"payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:21.544056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.543954 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/701abc1a-231d-42a0-a997-ba3e21759e37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:21.647370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.645169 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvgz\" (UniqueName: \"kubernetes.io/projected/701abc1a-231d-42a0-a997-ba3e21759e37-kube-api-access-ffvgz\") pod \"payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:21.647370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.645364 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/701abc1a-231d-42a0-a997-ba3e21759e37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:21.647370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.646307 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/701abc1a-231d-42a0-a997-ba3e21759e37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:21.670306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.670277 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvgz\" (UniqueName: \"kubernetes.io/projected/701abc1a-231d-42a0-a997-ba3e21759e37-kube-api-access-ffvgz\") pod \"payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:21.800845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:21.800759 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:22.028007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:22.027965 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" event={"ID":"b4d2e243-3b00-4807-8324-e14cb06f0d20","Type":"ContainerStarted","Data":"1b164e64e1d174dee9530f0a8d2b7f3926c62034d58ddab44c99c890f080f88c"} Nov 03 22:22:22.129766 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:22.129738 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq"] Nov 03 22:22:22.133847 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:22:22.133820 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701abc1a_231d_42a0_a997_ba3e21759e37.slice/crio-e43f24095af046add983cd1998f4c545430a5551bc407b66a99eff7ed0d533fe WatchSource:0}: Error finding container e43f24095af046add983cd1998f4c545430a5551bc407b66a99eff7ed0d533fe: Status 404 returned error can't find the container with id e43f24095af046add983cd1998f4c545430a5551bc407b66a99eff7ed0d533fe Nov 03 22:22:23.099207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:23.099161 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" event={"ID":"701abc1a-231d-42a0-a997-ba3e21759e37","Type":"ContainerStarted","Data":"e43f24095af046add983cd1998f4c545430a5551bc407b66a99eff7ed0d533fe"} Nov 03 22:22:23.956750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:23.956577 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq"] Nov 03 22:22:23.960501 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:23.960480 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:23.990818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:23.990788 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq"] Nov 03 22:22:23.991120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:23.990998 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5187e1-serving-cert\"" Nov 03 22:22:23.991120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:23.991007 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qn4p2\"" Nov 03 22:22:23.991120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:23.991093 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:22:24.009314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.009277 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.009503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.009342 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fac85f-2514-47c1-931a-37ccc7b40a6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.009572 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.009491 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxnt\" (UniqueName: \"kubernetes.io/projected/42fac85f-2514-47c1-931a-37ccc7b40a6e-kube-api-access-7kxnt\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.009632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.009570 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-metrics-certs\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.110217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.110185 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxnt\" (UniqueName: \"kubernetes.io/projected/42fac85f-2514-47c1-931a-37ccc7b40a6e-kube-api-access-7kxnt\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.110668 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.110248 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-metrics-certs\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.110668 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.110349 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.110668 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.110404 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fac85f-2514-47c1-931a-37ccc7b40a6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.110668 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:22:24.110610 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-serving-cert: secret "maas-api-e2e-worker-w3-5187e1-serving-cert" not found Nov 03 22:22:24.110925 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:22:24.110689 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls podName:42fac85f-2514-47c1-931a-37ccc7b40a6e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:24.610668469 +0000 UTC m=+566.040004441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls") pod "maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" (UID: "42fac85f-2514-47c1-931a-37ccc7b40a6e") : secret "maas-api-e2e-worker-w3-5187e1-serving-cert" not found Nov 03 22:22:24.111246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.111217 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fac85f-2514-47c1-931a-37ccc7b40a6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.111666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.111638 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" event={"ID":"b4d2e243-3b00-4807-8324-e14cb06f0d20","Type":"ContainerStarted","Data":"cb7707a1dcd6114da9ef7c8d1e095494aed9bf8effe0374ed5cfedb8b1cafc5f"} Nov 03 22:22:24.112422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.112366 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:24.113367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.113343 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-metrics-certs\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.164370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.164277 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxnt\" (UniqueName: \"kubernetes.io/projected/42fac85f-2514-47c1-931a-37ccc7b40a6e-kube-api-access-7kxnt\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.172934 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.170271 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" podStartSLOduration=1.443403469 podStartE2EDuration="4.17025072s" totalImagesPullingTime="2.726847251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:24.164560716 +0000 UTC m=+565.593896709" watchObservedRunningTime="2026-11-03 22:22:24.17025072 +0000 UTC m=+565.599586736" Nov 03 22:22:24.618131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.618086 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.671556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.670444 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:24.892466 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:24.892388 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:25.122195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.122154 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" event={"ID":"701abc1a-231d-42a0-a997-ba3e21759e37","Type":"ContainerStarted","Data":"5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450"} Nov 03 22:22:25.160653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.160556 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" podStartSLOduration=1.597346606 podStartE2EDuration="4.160536489s" totalImagesPullingTime="2.563189883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:25.159671603 +0000 UTC m=+566.589007598" watchObservedRunningTime="2026-11-03 22:22:25.160536489 +0000 UTC m=+566.589872483" Nov 03 22:22:25.205848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.205821 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq"] Nov 03 22:22:25.208331 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:22:25.208291 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fac85f_2514_47c1_931a_37ccc7b40a6e.slice/crio-d84368cf45b04daa9b583976fac67f5086b40a65c28b7d509ee94150509aca82 WatchSource:0}: Error finding container d84368cf45b04daa9b583976fac67f5086b40a65c28b7d509ee94150509aca82: Status 404 returned error can't find the container with id d84368cf45b04daa9b583976fac67f5086b40a65c28b7d509ee94150509aca82 Nov 03 22:22:25.481867 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.481769 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs"] Nov 03 22:22:25.487190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.487160 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:25.529497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.496829 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs"] Nov 03 22:22:25.529497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.512190 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-88a4d9\"" Nov 03 22:22:25.529497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.513000 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-88a4d9-dockercfg-9wmxp\"" Nov 03 22:22:25.655208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.655172 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltm82\" (UniqueName: \"kubernetes.io/projected/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-kube-api-access-ltm82\") pod \"payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:25.655208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.655211 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:25.757235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.756872 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltm82\" (UniqueName: \"kubernetes.io/projected/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-kube-api-access-ltm82\") pod \"payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:25.757235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.756937 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:25.757794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.757771 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:25.789752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.780087 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltm82\" (UniqueName: \"kubernetes.io/projected/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-kube-api-access-ltm82\") pod \"payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:25.866257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:25.865913 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:26.133294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:26.133164 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" event={"ID":"42fac85f-2514-47c1-931a-37ccc7b40a6e","Type":"ContainerStarted","Data":"d84368cf45b04daa9b583976fac67f5086b40a65c28b7d509ee94150509aca82"} Nov 03 22:22:26.133888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:26.133860 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:26.219357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:26.219326 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs"] Nov 03 22:22:27.147438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:27.147086 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" event={"ID":"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54","Type":"ContainerStarted","Data":"ddfba8271845e9ab73a90972e1c866c650a3156d0323f742a5d68abad11e2813"} Nov 03 22:22:27.147438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:27.147161 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" event={"ID":"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54","Type":"ContainerStarted","Data":"9b2ae4d17a10dbc4a3c52e2950e6b9e74e006d08eb2c8d96936f1b88b12fd9ea"} Nov 03 22:22:27.147438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:27.147411 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:27.187452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:27.187398 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" podStartSLOduration=2.187384233 podStartE2EDuration="2.187384233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:27.185300515 +0000 UTC m=+568.614636536" watchObservedRunningTime="2026-11-03 22:22:27.187384233 +0000 UTC m=+568.616720226" Nov 03 22:22:28.199090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:28.198777 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" event={"ID":"42fac85f-2514-47c1-931a-37ccc7b40a6e","Type":"ContainerStarted","Data":"60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e"} Nov 03 22:22:28.199571 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:28.199131 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:28.249193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:28.249131 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" podStartSLOduration=2.557219001 podStartE2EDuration="5.249115676s" totalImagesPullingTime="2.691896675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:28.24652187 +0000 UTC m=+569.675857874" watchObservedRunningTime="2026-11-03 22:22:28.249115676 +0000 UTC m=+569.678451669" Nov 03 22:22:32.186148 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:22:32.186107 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:22:32.905243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.905207 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp"] Nov 03 22:22:32.908764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.908709 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:32.956342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.956297 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-553753-serving-cert\"" Nov 03 22:22:32.974971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.974932 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp"] Nov 03 22:22:32.996425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.996387 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1996cfe-4286-48f4-9db9-93454203b234-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:32.996633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.996600 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-maas-api-tls\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:32.996765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.996745 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-metrics-certs\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:32.996872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:32.996858 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c29f\" (UniqueName: \"kubernetes.io/projected/b1996cfe-4286-48f4-9db9-93454203b234-kube-api-access-4c29f\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.098336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.098298 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-maas-api-tls\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.098526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.098395 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-metrics-certs\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.098526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.098421 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c29f\" (UniqueName: \"kubernetes.io/projected/b1996cfe-4286-48f4-9db9-93454203b234-kube-api-access-4c29f\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.098526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.098472 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1996cfe-4286-48f4-9db9-93454203b234-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.099249 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.099224 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1996cfe-4286-48f4-9db9-93454203b234-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.100852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.100830 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-maas-api-tls\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.100936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.100859 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-metrics-certs\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.120392 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.120351 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c29f\" (UniqueName: \"kubernetes.io/projected/b1996cfe-4286-48f4-9db9-93454203b234-kube-api-access-4c29f\") pod \"maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.221448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.221365 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:33.594902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:33.594503 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp"] Nov 03 22:22:34.316122 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:34.316033 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 22:22:34.319523 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:34.319492 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" event={"ID":"b1996cfe-4286-48f4-9db9-93454203b234","Type":"ContainerStarted","Data":"4893b6dd1ed7333f8a97f3b561cb4065f6d2fa337d021be29c2063c663556aa8"} Nov 03 22:22:34.319656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:34.319546 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" event={"ID":"b1996cfe-4286-48f4-9db9-93454203b234","Type":"ContainerStarted","Data":"5b668b1aef9074d69fc1e1373bb057f35a5812e0da046011e8218eb835376ea0"} Nov 03 22:22:34.319656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:34.319574 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:34.404044 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:34.403997 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" podStartSLOduration=2.058200946 podStartE2EDuration="2.403982891s" totalImagesPullingTime="345.781945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:34.402604586 +0000 UTC m=+575.831940571" watchObservedRunningTime="2026-11-03 22:22:34.403982891 +0000 UTC m=+575.833318882" Nov 03 22:22:36.161740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:36.159958 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 22:22:37.170808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:37.169785 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 22:22:38.203758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:38.202527 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 22:22:40.360738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:40.350889 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 22:22:41.436172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:41.434870 4625 generic.go:356] "Generic (PLEG): container finished" podID="2af5050e-5e51-4d60-a172-64c2d154a063" containerID="659a948bb9b9529f05c14af772ecb97f1f721a452308f4463928d83ea49b7abe" exitCode=0 Nov 03 22:22:41.436172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:41.434994 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" event={"ID":"2af5050e-5e51-4d60-a172-64c2d154a063","Type":"ContainerDied","Data":"659a948bb9b9529f05c14af772ecb97f1f721a452308f4463928d83ea49b7abe"} Nov 03 22:22:41.436172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:41.435437 4625 scope.go:122] "RemoveContainer" containerID="659a948bb9b9529f05c14af772ecb97f1f721a452308f4463928d83ea49b7abe" Nov 03 22:22:41.437976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:41.437936 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:41.438099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:41.438031 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:41.438099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:41.438077 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:42.467805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:42.467762 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" event={"ID":"2af5050e-5e51-4d60-a172-64c2d154a063","Type":"ContainerStarted","Data":"a83b3eee866a98867879cd0d2ac5261fd3025ee138191d39a2d11bb2f516e7fb"} Nov 03 22:22:42.971982 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:42.971950 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:42.973220 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:42.973191 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:22:42.973287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:42.973257 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 22:22:43.976485 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:43.976250 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:22:43.976485 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:43.976311 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 22:22:44.984048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:44.981024 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:44.998273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:44.997290 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:22:44.998273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:44.997350 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 22:22:45.025033 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.024900 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:22:45.025033 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.024960 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:22:45.128446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.127974 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:37752->10.134.0.57:15021: read: connection reset by peer" start-of-body= Nov 03 22:22:45.128446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.128024 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:37752->10.134.0.57:15021: read: connection reset by peer" Nov 03 22:22:45.504198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.504163 4625 generic.go:356] "Generic (PLEG): container finished" podID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerID="9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0" exitCode=0 Nov 03 22:22:45.504390 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.504264 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" event={"ID":"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43","Type":"ContainerDied","Data":"9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0"} Nov 03 22:22:45.504646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.504630 4625 scope.go:122] "RemoveContainer" containerID="9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0" Nov 03 22:22:45.506188 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.506154 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:45.506264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.506245 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:45.506304 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.506281 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:45.979740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.972167 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:22:45.979740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:45.972232 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 22:22:46.636741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:46.636339 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" event={"ID":"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43","Type":"ContainerStarted","Data":"28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1"} Nov 03 22:22:46.973232 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:46.973134 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:22:46.973532 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:46.973497 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 22:22:47.280800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.277268 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:47.286957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.285036 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:22:47.286957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.285096 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 22:22:47.663753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.663204 4625 generic.go:356] "Generic (PLEG): container finished" podID="e6725a80-3ec7-436a-a031-240f2c83e623" containerID="2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80" exitCode=0 Nov 03 22:22:47.666304 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.664693 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" event={"ID":"e6725a80-3ec7-436a-a031-240f2c83e623","Type":"ContainerDied","Data":"2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80"} Nov 03 22:22:47.666304 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.665126 4625 scope.go:122] "RemoveContainer" containerID="2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80" Nov 03 22:22:47.676806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.667671 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:47.676806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.667802 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:47.676806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:47.667852 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:48.282790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:48.281855 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:22:48.282790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:48.281933 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 22:22:48.291648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:48.290313 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:48.294207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:48.294185 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:22:48.702019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:48.701984 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" event={"ID":"e6725a80-3ec7-436a-a031-240f2c83e623","Type":"ContainerStarted","Data":"fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7"} Nov 03 22:22:49.275741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:49.275233 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:22:49.275741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:49.275294 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 22:22:49.382131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:49.382096 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:49.382518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:49.382420 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:22:49.382699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:49.382673 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 22:22:50.288790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:50.275124 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:22:50.288790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:50.275187 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 22:22:50.380179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:50.380134 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:22:50.380354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:50.380203 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 22:22:51.275555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:51.275110 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:22:51.275555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:51.275173 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 22:22:51.383751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:51.383123 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:22:51.383751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:51.383183 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 22:22:52.379486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:52.379438 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:22:52.379701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:52.379507 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 22:22:52.406012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:52.405979 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:52.406605 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:52.406567 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:52.407766 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:52.407693 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:22:53.379881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:53.379843 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:22:53.380054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:53.379905 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 22:22:54.379379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:54.379348 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:22:54.379816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:54.379400 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 22:22:55.383001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.382920 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:55.383519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.383329 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:55.383943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.383924 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 22:22:55.826687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.826602 4625 generic.go:356] "Generic (PLEG): container finished" podID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerID="252d83aac6558a86207e4b9d8a14ba113749662c3957dd83dbcc913a59e02bcd" exitCode=0 Nov 03 22:22:55.826687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.826671 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" event={"ID":"f225f6da-aad0-43cb-9fe8-6fab90bbf69b","Type":"ContainerDied","Data":"252d83aac6558a86207e4b9d8a14ba113749662c3957dd83dbcc913a59e02bcd"} Nov 03 22:22:55.827175 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.827160 4625 scope.go:122] "RemoveContainer" containerID="252d83aac6558a86207e4b9d8a14ba113749662c3957dd83dbcc913a59e02bcd" Nov 03 22:22:55.828504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.828477 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:55.828574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.828545 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:55.828614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:55.828579 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:22:56.878743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:56.864531 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" event={"ID":"f225f6da-aad0-43cb-9fe8-6fab90bbf69b","Type":"ContainerStarted","Data":"733e9ad661b343ba6de35c74c814380734c6c2d265674ab58e39aac85c1e007b"} Nov 03 22:22:57.543508 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:57.543280 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:22:57.544991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:57.544962 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:22:57.545220 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:57.545191 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 22:22:58.558043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:58.558008 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:22:58.565257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:58.565210 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 22:22:58.801919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:58.801885 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:22:58.804286 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:58.804262 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:22:59.546864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:59.546831 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:22:59.547152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:59.547114 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 22:22:59.981452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:22:59.981403 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:23:00.543848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:00.543809 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:23:00.544164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:00.544130 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 22:23:01.544740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:01.541876 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:23:01.544740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:01.541951 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 22:23:02.185439 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:23:02.185401 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:23:02.544756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:02.544649 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:23:02.545741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:02.545700 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:23:32.188448 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:23:32.188401 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:23:58.120869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.120822 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc"] Nov 03 22:23:58.124996 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.124979 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.143353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.143325 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e19850-openshift-default-dockercfg-5fvb8\"" Nov 03 22:23:58.156788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.156759 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc"] Nov 03 22:23:58.224788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.224752 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.224962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.224859 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.224962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.224928 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.225074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.224966 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.225074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.225007 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.225185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.225114 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.225185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.225156 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.225287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.225247 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.225340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.225301 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b5jh\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-kube-api-access-9b5jh\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326417 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b5jh\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-kube-api-access-9b5jh\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326495 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326560 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326603 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326642 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326685 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326754 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326784 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.326924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.326891 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.327140 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.327080 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.327264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.327241 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.327328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.327282 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.327328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.327314 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.327706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.327681 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.328883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.328860 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.329396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.329372 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.351351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.351320 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.351495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.351401 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b5jh\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-kube-api-access-9b5jh\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.438574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.438494 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:58.779172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.779143 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc"] Nov 03 22:23:58.781208 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:23:58.781180 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4eade4_f7c3_4160_a9c4_2c23dfb73db7.slice/crio-431916a897042ccc9e1fb3c967825ef1f1bab518990b21bc48cd1b22ac9d5c63 WatchSource:0}: Error finding container 431916a897042ccc9e1fb3c967825ef1f1bab518990b21bc48cd1b22ac9d5c63: Status 404 returned error can't find the container with id 431916a897042ccc9e1fb3c967825ef1f1bab518990b21bc48cd1b22ac9d5c63 Nov 03 22:23:58.783671 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.783641 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:23:58.783770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.783702 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:23:58.783770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:58.783756 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:23:59.269739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:59.268687 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" event={"ID":"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7","Type":"ContainerStarted","Data":"686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212"} Nov 03 22:23:59.269739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:59.268835 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" event={"ID":"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7","Type":"ContainerStarted","Data":"431916a897042ccc9e1fb3c967825ef1f1bab518990b21bc48cd1b22ac9d5c63"} Nov 03 22:23:59.313970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:59.313917 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podStartSLOduration=1.313902665 podStartE2EDuration="1.313902665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:59.312023238 +0000 UTC m=+660.741359232" watchObservedRunningTime="2026-11-03 22:23:59.313902665 +0000 UTC m=+660.743238649" Nov 03 22:23:59.443828 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:59.442073 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:23:59.453740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:59.445823 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:23:59.453740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:23:59.445886 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:00.445148 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:00.445110 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:00.445836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:00.445802 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:01.443746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:01.443119 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:01.443746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:01.443181 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:02.194453 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:24:02.194413 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:24:02.450802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:02.443108 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:02.450802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:02.443173 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:03.440740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:03.439104 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:03.440740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:03.439163 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:04.460862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:04.450854 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:04.460862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:04.450942 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:05.874095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:05.874059 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:24:05.875387 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:05.875244 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:24:05.876020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:05.875993 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:24:32.188196 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:24:32.188159 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:24:32.324017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:32.323975 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" podUID="7107b00f-9a1b-4b01-a596-b450fa81b1f4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.39:8081/readyz\": dial tcp 10.134.0.39:8081: connect: connection refused" Nov 03 22:24:32.528942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:32.528849 4625 generic.go:356] "Generic (PLEG): container finished" podID="7107b00f-9a1b-4b01-a596-b450fa81b1f4" containerID="ed622a5a7836e2cbc4d521d835725899c344b71e639ef32117e3f955f93f995b" exitCode=1 Nov 03 22:24:32.529116 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:32.528935 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" event={"ID":"7107b00f-9a1b-4b01-a596-b450fa81b1f4","Type":"ContainerDied","Data":"ed622a5a7836e2cbc4d521d835725899c344b71e639ef32117e3f955f93f995b"} Nov 03 22:24:32.529366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:32.529350 4625 scope.go:122] "RemoveContainer" containerID="ed622a5a7836e2cbc4d521d835725899c344b71e639ef32117e3f955f93f995b" Nov 03 22:24:33.538098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:33.538064 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" event={"ID":"7107b00f-9a1b-4b01-a596-b450fa81b1f4","Type":"ContainerStarted","Data":"52379ce741ba6595f96b99cf133e0e9f7b8329a21a4e98793b034fbc6e567b61"} Nov 03 22:24:33.538599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:33.538292 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:24:44.549250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:24:44.549213 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xn8cx" Nov 03 22:25:02.190424 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:25:02.190384 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:25:15.654411 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.654370 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb"] Nov 03 22:25:15.658369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.658349 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.661551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.661517 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbs6\" (UniqueName: \"kubernetes.io/projected/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-kube-api-access-vgbs6\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.661701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.661627 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.661701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.661663 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.683125 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.683091 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb"] Nov 03 22:25:15.689367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.689339 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:15.689518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.689393 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:15.762401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.762365 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbs6\" (UniqueName: \"kubernetes.io/projected/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-kube-api-access-vgbs6\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.762627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.762408 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.762627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.762430 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.763176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.763148 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.765048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.765017 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.785040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.785007 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbs6\" (UniqueName: \"kubernetes.io/projected/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-kube-api-access-vgbs6\") pod \"maas-api-revoke-sub-59ec3023-t25zb\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:15.975729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:15.975608 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:16.282168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:16.282133 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb"] Nov 03 22:25:16.284118 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:25:16.284089 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceef80a8_ad4e_4df9_afcd_7f8f4d46fe8d.slice/crio-77b264d16fb1b2701553b861fa64e8f5e576ae1790bca2700ec99e9ae5c35af2 WatchSource:0}: Error finding container 77b264d16fb1b2701553b861fa64e8f5e576ae1790bca2700ec99e9ae5c35af2: Status 404 returned error can't find the container with id 77b264d16fb1b2701553b861fa64e8f5e576ae1790bca2700ec99e9ae5c35af2 Nov 03 22:25:16.286542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:16.286518 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:25:16.817590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:16.817553 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" event={"ID":"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d","Type":"ContainerStarted","Data":"77b264d16fb1b2701553b861fa64e8f5e576ae1790bca2700ec99e9ae5c35af2"} Nov 03 22:25:19.841224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:19.841186 4625 generic.go:356] "Generic (PLEG): container finished" podID="ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" containerID="d24d2360725745d6200bac1f34ff89b69b933c971856e92869f08da8ae291ba9" exitCode=0 Nov 03 22:25:19.841705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:19.841269 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" event={"ID":"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d","Type":"ContainerDied","Data":"d24d2360725745d6200bac1f34ff89b69b933c971856e92869f08da8ae291ba9"} Nov 03 22:25:21.023931 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.023900 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:21.125013 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.124930 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-xks-ca\") pod \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " Nov 03 22:25:21.125013 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.124999 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-openshift-ca\") pod \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " Nov 03 22:25:21.125250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.125141 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-kube-api-access-vgbs6\" (UniqueName: \"kubernetes.io/projected/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-kube-api-access-vgbs6\") pod \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\" (UID: \"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d\") " Nov 03 22:25:21.125422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.125395 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-openshift-ca" pod "ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" (UID: "ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:21.125566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.125549 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:21.127463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.127439 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-kube-api-access-vgbs6" pod "ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" (UID: "ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d"). InnerVolumeSpecName "kube-api-access-vgbs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:21.127463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.127451 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-xks-ca" pod "ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" (UID: "ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:21.229576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.229499 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbs6\" (UniqueName: \"kubernetes.io/projected/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-kube-api-access-vgbs6\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:21.229576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.229541 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:21.859081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.859047 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" Nov 03 22:25:21.859292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.859054 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb" event={"ID":"ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d","Type":"ContainerDied","Data":"77b264d16fb1b2701553b861fa64e8f5e576ae1790bca2700ec99e9ae5c35af2"} Nov 03 22:25:21.859292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:21.859179 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77b264d16fb1b2701553b861fa64e8f5e576ae1790bca2700ec99e9ae5c35af2" Nov 03 22:25:23.314910 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:23.314856 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:25:23.315450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:23.314936 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:25:32.200742 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:25:32.200118 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:25:36.876255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.876193 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms"] Nov 03 22:25:36.876786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.876730 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" containerName="revoke-keys" Nov 03 22:25:36.876786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.876743 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" containerName="revoke-keys" Nov 03 22:25:36.876942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.876847 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" containerName="revoke-keys" Nov 03 22:25:36.884097 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.884074 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:36.898888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.898859 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms"] Nov 03 22:25:36.954344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.954319 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:36.954496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.954416 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:36.982132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.982104 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:36.982276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.982148 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwg8\" (UniqueName: \"kubernetes.io/projected/1004577a-94a3-42e3-a224-c3114fb07b33-kube-api-access-5bwg8\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:36.982318 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:36.982267 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.083186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.083152 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.083186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.083193 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwg8\" (UniqueName: \"kubernetes.io/projected/1004577a-94a3-42e3-a224-c3114fb07b33-kube-api-access-5bwg8\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.083389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.083240 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.083998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.083978 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.085590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.085572 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.105068 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.105041 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwg8\" (UniqueName: \"kubernetes.io/projected/1004577a-94a3-42e3-a224-c3114fb07b33-kube-api-access-5bwg8\") pod \"maas-api-revoke-sub-56f651cd-hscms\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.200791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.200688 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:37.514798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:37.514774 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms"] Nov 03 22:25:37.516526 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:25:37.516499 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1004577a_94a3_42e3_a224_c3114fb07b33.slice/crio-ba45ef0dee1fec1f0ca1fbd84e3a2ff5ec94b1f0c79b978e006a0c59b3e9c09c WatchSource:0}: Error finding container ba45ef0dee1fec1f0ca1fbd84e3a2ff5ec94b1f0c79b978e006a0c59b3e9c09c: Status 404 returned error can't find the container with id ba45ef0dee1fec1f0ca1fbd84e3a2ff5ec94b1f0c79b978e006a0c59b3e9c09c Nov 03 22:25:38.013041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.013006 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" event={"ID":"1004577a-94a3-42e3-a224-c3114fb07b33","Type":"ContainerStarted","Data":"0aba23a7d2d31502bd0579702588e0adf506ccbb8e25dbcea12bf9963f16125a"} Nov 03 22:25:38.013537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.013059 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" event={"ID":"1004577a-94a3-42e3-a224-c3114fb07b33","Type":"ContainerStarted","Data":"ba45ef0dee1fec1f0ca1fbd84e3a2ff5ec94b1f0c79b978e006a0c59b3e9c09c"} Nov 03 22:25:38.055299 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.055236 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" podStartSLOduration=2.055211138 podStartE2EDuration="2.055211138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:38.050441591 +0000 UTC m=+759.479777589" watchObservedRunningTime="2026-11-03 22:25:38.055211138 +0000 UTC m=+759.484547133" Nov 03 22:25:38.179725 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.179635 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg"] Nov 03 22:25:38.183724 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.183701 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.230753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.228883 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg"] Nov 03 22:25:38.296426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.296386 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.296613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.296507 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstjw\" (UniqueName: \"kubernetes.io/projected/c599fd8d-3bfc-4382-862e-7eab9399e89b-kube-api-access-wstjw\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.296613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.296527 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.397604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.397567 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.397796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.397652 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstjw\" (UniqueName: \"kubernetes.io/projected/c599fd8d-3bfc-4382-862e-7eab9399e89b-kube-api-access-wstjw\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.397796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.397677 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.398209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.398183 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.400106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.400077 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.420055 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.420029 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstjw\" (UniqueName: \"kubernetes.io/projected/c599fd8d-3bfc-4382-862e-7eab9399e89b-kube-api-access-wstjw\") pod \"maas-api-revoke-sub-4502c7d3-n6nfg\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.497772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.497664 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:38.806757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:38.806732 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg"] Nov 03 22:25:38.808692 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:25:38.808665 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc599fd8d_3bfc_4382_862e_7eab9399e89b.slice/crio-b5bc6a070637b987873b2cc7078d5ec028415d38e2603de6f23b1aff89a02872 WatchSource:0}: Error finding container b5bc6a070637b987873b2cc7078d5ec028415d38e2603de6f23b1aff89a02872: Status 404 returned error can't find the container with id b5bc6a070637b987873b2cc7078d5ec028415d38e2603de6f23b1aff89a02872 Nov 03 22:25:39.020696 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:39.020661 4625 generic.go:356] "Generic (PLEG): container finished" podID="1004577a-94a3-42e3-a224-c3114fb07b33" containerID="0aba23a7d2d31502bd0579702588e0adf506ccbb8e25dbcea12bf9963f16125a" exitCode=0 Nov 03 22:25:39.021165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:39.020749 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" event={"ID":"1004577a-94a3-42e3-a224-c3114fb07b33","Type":"ContainerDied","Data":"0aba23a7d2d31502bd0579702588e0adf506ccbb8e25dbcea12bf9963f16125a"} Nov 03 22:25:39.022158 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:39.022137 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" event={"ID":"c599fd8d-3bfc-4382-862e-7eab9399e89b","Type":"ContainerStarted","Data":"b5bc6a070637b987873b2cc7078d5ec028415d38e2603de6f23b1aff89a02872"} Nov 03 22:25:40.030475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.030374 4625 generic.go:356] "Generic (PLEG): container finished" podID="c599fd8d-3bfc-4382-862e-7eab9399e89b" containerID="0fdbf9b9dcd6b5b2940dc9059e80caf6bc25f08c2eb8061913d26135c29776bd" exitCode=0 Nov 03 22:25:40.030928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.030480 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" event={"ID":"c599fd8d-3bfc-4382-862e-7eab9399e89b","Type":"ContainerDied","Data":"0fdbf9b9dcd6b5b2940dc9059e80caf6bc25f08c2eb8061913d26135c29776bd"} Nov 03 22:25:40.207815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.207786 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:40.318685 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.318588 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-xks-ca\") pod \"1004577a-94a3-42e3-a224-c3114fb07b33\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " Nov 03 22:25:40.318685 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.318658 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-openshift-ca\") pod \"1004577a-94a3-42e3-a224-c3114fb07b33\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " Nov 03 22:25:40.318942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.318791 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1004577a-94a3-42e3-a224-c3114fb07b33-kube-api-access-5bwg8\" (UniqueName: \"kubernetes.io/projected/1004577a-94a3-42e3-a224-c3114fb07b33-kube-api-access-5bwg8\") pod \"1004577a-94a3-42e3-a224-c3114fb07b33\" (UID: \"1004577a-94a3-42e3-a224-c3114fb07b33\") " Nov 03 22:25:40.319085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.319059 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-openshift-ca" pod "1004577a-94a3-42e3-a224-c3114fb07b33" (UID: "1004577a-94a3-42e3-a224-c3114fb07b33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:40.319270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.319251 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.320875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.320854 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1004577a-94a3-42e3-a224-c3114fb07b33-kube-api-access-5bwg8" pod "1004577a-94a3-42e3-a224-c3114fb07b33" (UID: "1004577a-94a3-42e3-a224-c3114fb07b33"). InnerVolumeSpecName "kube-api-access-5bwg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:40.321199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.321167 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-xks-ca" pod "1004577a-94a3-42e3-a224-c3114fb07b33" (UID: "1004577a-94a3-42e3-a224-c3114fb07b33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:40.420827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.420782 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwg8\" (UniqueName: \"kubernetes.io/projected/1004577a-94a3-42e3-a224-c3114fb07b33-kube-api-access-5bwg8\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.420827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:40.420823 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1004577a-94a3-42e3-a224-c3114fb07b33-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:41.038770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.038648 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" Nov 03 22:25:41.038770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.038666 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms" event={"ID":"1004577a-94a3-42e3-a224-c3114fb07b33","Type":"ContainerDied","Data":"ba45ef0dee1fec1f0ca1fbd84e3a2ff5ec94b1f0c79b978e006a0c59b3e9c09c"} Nov 03 22:25:41.038770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.038730 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba45ef0dee1fec1f0ca1fbd84e3a2ff5ec94b1f0c79b978e006a0c59b3e9c09c" Nov 03 22:25:41.176852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.176828 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:41.240638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.240598 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-xks-ca\") pod \"c599fd8d-3bfc-4382-862e-7eab9399e89b\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " Nov 03 22:25:41.240864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.240674 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c599fd8d-3bfc-4382-862e-7eab9399e89b-kube-api-access-wstjw\" (UniqueName: \"kubernetes.io/projected/c599fd8d-3bfc-4382-862e-7eab9399e89b-kube-api-access-wstjw\") pod \"c599fd8d-3bfc-4382-862e-7eab9399e89b\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " Nov 03 22:25:41.240864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.240797 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-openshift-ca\") pod \"c599fd8d-3bfc-4382-862e-7eab9399e89b\" (UID: \"c599fd8d-3bfc-4382-862e-7eab9399e89b\") " Nov 03 22:25:41.241968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.241938 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-openshift-ca" pod "c599fd8d-3bfc-4382-862e-7eab9399e89b" (UID: "c599fd8d-3bfc-4382-862e-7eab9399e89b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:41.245047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.244706 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-xks-ca" pod "c599fd8d-3bfc-4382-862e-7eab9399e89b" (UID: "c599fd8d-3bfc-4382-862e-7eab9399e89b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:41.247257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.247215 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c599fd8d-3bfc-4382-862e-7eab9399e89b-kube-api-access-wstjw" pod "c599fd8d-3bfc-4382-862e-7eab9399e89b" (UID: "c599fd8d-3bfc-4382-862e-7eab9399e89b"). InnerVolumeSpecName "kube-api-access-wstjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:41.346368 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.345260 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstjw\" (UniqueName: \"kubernetes.io/projected/c599fd8d-3bfc-4382-862e-7eab9399e89b-kube-api-access-wstjw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:41.346368 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.345302 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:41.346368 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:41.345318 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c599fd8d-3bfc-4382-862e-7eab9399e89b-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:42.047508 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:42.047469 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" event={"ID":"c599fd8d-3bfc-4382-862e-7eab9399e89b","Type":"ContainerDied","Data":"b5bc6a070637b987873b2cc7078d5ec028415d38e2603de6f23b1aff89a02872"} Nov 03 22:25:42.047932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:42.047527 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5bc6a070637b987873b2cc7078d5ec028415d38e2603de6f23b1aff89a02872" Nov 03 22:25:42.047932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:42.047557 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg" Nov 03 22:25:44.150977 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.150938 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s"] Nov 03 22:25:44.151743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.151691 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c599fd8d-3bfc-4382-862e-7eab9399e89b" containerName="revoke-keys" Nov 03 22:25:44.151743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.151735 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c599fd8d-3bfc-4382-862e-7eab9399e89b" containerName="revoke-keys" Nov 03 22:25:44.151845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.151836 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1004577a-94a3-42e3-a224-c3114fb07b33" containerName="revoke-keys" Nov 03 22:25:44.151888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.151847 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1004577a-94a3-42e3-a224-c3114fb07b33" containerName="revoke-keys" Nov 03 22:25:44.151985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.151971 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1004577a-94a3-42e3-a224-c3114fb07b33" containerName="revoke-keys" Nov 03 22:25:44.152020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.151996 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c599fd8d-3bfc-4382-862e-7eab9399e89b" containerName="revoke-keys" Nov 03 22:25:44.157661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.157635 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.180300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.180265 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s"] Nov 03 22:25:44.279921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.279887 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.279932 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e5926-471e-4f07-ae43-c988b094340c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.280064 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.280113 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz6pl\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-kube-api-access-bz6pl\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.280157 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e5926-471e-4f07-ae43-c988b094340c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.280180 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.280274 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.280316 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.280339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.280340 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382321 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382391 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382418 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382443 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382581 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e5926-471e-4f07-ae43-c988b094340c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382679 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382767 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6pl\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-kube-api-access-bz6pl\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382820 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382870 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e5926-471e-4f07-ae43-c988b094340c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382887 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382903 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.382991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.382926 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.383187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.383150 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.383448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.383426 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e5926-471e-4f07-ae43-c988b094340c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.385136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.385112 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.385413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.385395 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e5926-471e-4f07-ae43-c988b094340c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.404830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.404766 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.404955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.404904 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6pl\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-kube-api-access-bz6pl\") pod \"maas-default-gateway-openshift-default-7df854bf67-ddx5s\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.476576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.476541 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:44.854480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.854444 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s"] Nov 03 22:25:44.892292 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:25:44.892251 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61e5926_471e_4f07_ae43_c988b094340c.slice/crio-2facbd464da16610e305d4bd9671bd725ad9a5163f880d4b6d311eb229086036 WatchSource:0}: Error finding container 2facbd464da16610e305d4bd9671bd725ad9a5163f880d4b6d311eb229086036: Status 404 returned error can't find the container with id 2facbd464da16610e305d4bd9671bd725ad9a5163f880d4b6d311eb229086036 Nov 03 22:25:44.913862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.913822 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:44.913985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.913895 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:44.913985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:44.913927 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:45.136833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:45.129509 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" event={"ID":"b61e5926-471e-4f07-ae43-c988b094340c","Type":"ContainerStarted","Data":"4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3"} Nov 03 22:25:45.136833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:45.129596 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" event={"ID":"b61e5926-471e-4f07-ae43-c988b094340c","Type":"ContainerStarted","Data":"2facbd464da16610e305d4bd9671bd725ad9a5163f880d4b6d311eb229086036"} Nov 03 22:25:45.482796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:45.479366 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:45.482796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:45.479494 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:45.482796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:45.479541 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:46.483239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:46.483193 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:46.484039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:46.484000 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:47.477770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:47.477706 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:47.478056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:47.477795 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:48.477994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:48.477962 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:48.478492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:48.478014 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:49.477518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:49.477479 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:49.477756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:49.477544 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:50.477262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.477227 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:50.477658 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.477279 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:50.776527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.776417 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podStartSLOduration=6.776396273 podStartE2EDuration="6.776396273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:45.181532261 +0000 UTC m=+766.610868255" watchObservedRunningTime="2026-11-03 22:25:50.776396273 +0000 UTC m=+772.205732267" Nov 03 22:25:50.777287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.777268 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4"] Nov 03 22:25:50.781809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.781784 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.803833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.803797 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4"] Nov 03 22:25:50.806844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.806814 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.806983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.806949 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.807051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.807020 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8sx\" (UniqueName: \"kubernetes.io/projected/0e34492c-6b5f-441b-b240-99ac9d5b58e1-kube-api-access-5l8sx\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.809039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.809004 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:50.809163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.809045 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:50.908742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.908693 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.908955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.908787 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.908955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.908851 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8sx\" (UniqueName: \"kubernetes.io/projected/0e34492c-6b5f-441b-b240-99ac9d5b58e1-kube-api-access-5l8sx\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.909463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.909437 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.911466 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.911435 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:50.931746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:50.931689 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8sx\" (UniqueName: \"kubernetes.io/projected/0e34492c-6b5f-441b-b240-99ac9d5b58e1-kube-api-access-5l8sx\") pod \"maas-api-revoke-sub-af37c0f4-q6bb4\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:51.098178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:51.098140 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:51.455383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:51.455346 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4"] Nov 03 22:25:51.477492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:51.477463 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:51.477876 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:51.477528 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:52.304359 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:52.304295 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" event={"ID":"0e34492c-6b5f-441b-b240-99ac9d5b58e1","Type":"ContainerStarted","Data":"33f83aa8aea956c3bc0dec85fd7f27c5041a5230f8a51b5a2b9b6dd6b6f15799"} Nov 03 22:25:52.304564 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:52.304415 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" event={"ID":"0e34492c-6b5f-441b-b240-99ac9d5b58e1","Type":"ContainerStarted","Data":"55d9686741e7d7d32d3d6640057e75cee52882965d9bbc72171dd9acc633304b"} Nov 03 22:25:52.345316 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:52.345256 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" podStartSLOduration=2.345237042 podStartE2EDuration="2.345237042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:52.342708782 +0000 UTC m=+773.772044776" watchObservedRunningTime="2026-11-03 22:25:52.345237042 +0000 UTC m=+773.774573036" Nov 03 22:25:52.477801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:52.477760 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:25:52.478214 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:52.477844 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:25:53.315988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.315946 4625 generic.go:356] "Generic (PLEG): container finished" podID="0e34492c-6b5f-441b-b240-99ac9d5b58e1" containerID="33f83aa8aea956c3bc0dec85fd7f27c5041a5230f8a51b5a2b9b6dd6b6f15799" exitCode=0 Nov 03 22:25:53.316199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.316043 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" event={"ID":"0e34492c-6b5f-441b-b240-99ac9d5b58e1","Type":"ContainerDied","Data":"33f83aa8aea956c3bc0dec85fd7f27c5041a5230f8a51b5a2b9b6dd6b6f15799"} Nov 03 22:25:53.424483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.424443 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2"] Nov 03 22:25:53.429491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.429461 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.443639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.443610 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"distinct-w0-88a4d9-kserve-self-signed-certs\"" Nov 03 22:25:53.445215 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.445194 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"default-dockercfg-kw52f\"" Nov 03 22:25:53.446230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.446204 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"kube-root-ca.crt\"" Nov 03 22:25:53.446336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.446279 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"openshift-service-ca.crt\"" Nov 03 22:25:53.447138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.447108 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2"] Nov 03 22:25:53.548341 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.548299 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7h9\" (UniqueName: \"kubernetes.io/projected/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-kube-api-access-9s7h9\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.548829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.548681 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-dshm\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.548829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.548747 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-model-cache\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.548957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.548836 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-home\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.548957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.548906 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tls-certs\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.548957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.548934 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tmp-dir\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.651389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.651342 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-home\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.651699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.651679 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tls-certs\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.651857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.651839 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tmp-dir\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.652079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.652062 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7h9\" (UniqueName: \"kubernetes.io/projected/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-kube-api-access-9s7h9\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.652228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.652211 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-dshm\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.652379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.652342 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-model-cache\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.652983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.652962 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-model-cache\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.653567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.653545 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tmp-dir\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.654192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.653813 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-home\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.668166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.668081 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-dshm\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.668547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.668520 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tls-certs\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.693452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.693384 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7h9\" (UniqueName: \"kubernetes.io/projected/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-kube-api-access-9s7h9\") pod \"distinct-w0-88a4d9-kserve-84b7f99547-bktt2\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.750264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.750227 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:53.979335 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.979255 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:53.979574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.979550 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:53.980575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:53.980552 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:25:54.096456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.096414 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2"] Nov 03 22:25:54.105065 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:25:54.105024 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d2a6a8c_1a88_4ecf_9a48_f06b18e51858.slice/crio-cd5c0f110dc39e7a48c190fdb223392b0d63db04b1e4bd375ff2dcb479e28304 WatchSource:0}: Error finding container cd5c0f110dc39e7a48c190fdb223392b0d63db04b1e4bd375ff2dcb479e28304: Status 404 returned error can't find the container with id cd5c0f110dc39e7a48c190fdb223392b0d63db04b1e4bd375ff2dcb479e28304 Nov 03 22:25:54.323974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.323879 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" event={"ID":"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858","Type":"ContainerStarted","Data":"a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9"} Nov 03 22:25:54.323974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.323930 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" event={"ID":"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858","Type":"ContainerStarted","Data":"cd5c0f110dc39e7a48c190fdb223392b0d63db04b1e4bd375ff2dcb479e28304"} Nov 03 22:25:54.365743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.365650 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podStartSLOduration=1.365631236 podStartE2EDuration="1.365631236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:54.363967653 +0000 UTC m=+775.793303650" watchObservedRunningTime="2026-11-03 22:25:54.365631236 +0000 UTC m=+775.794967231" Nov 03 22:25:54.457878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.457849 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:54.579261 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.579161 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e34492c-6b5f-441b-b240-99ac9d5b58e1-kube-api-access-5l8sx\" (UniqueName: \"kubernetes.io/projected/0e34492c-6b5f-441b-b240-99ac9d5b58e1-kube-api-access-5l8sx\") pod \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " Nov 03 22:25:54.579849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.579269 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-openshift-ca\") pod \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " Nov 03 22:25:54.579849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.579431 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-xks-ca\") pod \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\" (UID: \"0e34492c-6b5f-441b-b240-99ac9d5b58e1\") " Nov 03 22:25:54.580901 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.580867 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-openshift-ca" pod "0e34492c-6b5f-441b-b240-99ac9d5b58e1" (UID: "0e34492c-6b5f-441b-b240-99ac9d5b58e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:54.582393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.582363 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-xks-ca" pod "0e34492c-6b5f-441b-b240-99ac9d5b58e1" (UID: "0e34492c-6b5f-441b-b240-99ac9d5b58e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:54.583829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.583790 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e34492c-6b5f-441b-b240-99ac9d5b58e1-kube-api-access-5l8sx" pod "0e34492c-6b5f-441b-b240-99ac9d5b58e1" (UID: "0e34492c-6b5f-441b-b240-99ac9d5b58e1"). InnerVolumeSpecName "kube-api-access-5l8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:54.680662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.680629 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:54.680662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.680659 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8sx\" (UniqueName: \"kubernetes.io/projected/0e34492c-6b5f-441b-b240-99ac9d5b58e1-kube-api-access-5l8sx\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:54.680880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.680670 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e34492c-6b5f-441b-b240-99ac9d5b58e1-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:25:54.750627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:54.750600 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:25:55.332669 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:55.332641 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" Nov 03 22:25:55.332875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:55.332679 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4" event={"ID":"0e34492c-6b5f-441b-b240-99ac9d5b58e1","Type":"ContainerDied","Data":"55d9686741e7d7d32d3d6640057e75cee52882965d9bbc72171dd9acc633304b"} Nov 03 22:25:55.332875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:55.332760 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d9686741e7d7d32d3d6640057e75cee52882965d9bbc72171dd9acc633304b" Nov 03 22:25:57.924935 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:57.924904 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6"] Nov 03 22:25:57.925442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:57.925426 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0e34492c-6b5f-441b-b240-99ac9d5b58e1" containerName="revoke-keys" Nov 03 22:25:57.925486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:57.925444 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e34492c-6b5f-441b-b240-99ac9d5b58e1" containerName="revoke-keys" Nov 03 22:25:57.925540 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:57.925531 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e34492c-6b5f-441b-b240-99ac9d5b58e1" containerName="revoke-keys" Nov 03 22:25:57.928777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:57.928759 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:57.957272 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:57.956175 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6"] Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.022264 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd"] Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.023344 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.023385 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.024087 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ada526-3272-44d6-a5ba-276199cb022b-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.024128 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.024153 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ada526-3272-44d6-a5ba-276199cb022b-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.024194 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.024278 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwsf\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-kube-api-access-kqwsf\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.024312 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.029730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.024434 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.031129 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.031101 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.051221 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.051195 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd"] Nov 03 22:25:58.057744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.057695 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2d619f-openshift-default-dockercfg-2f4cd\"" Nov 03 22:25:58.126243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126200 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.126502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126258 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shnbz\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-kube-api-access-shnbz\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.126502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126318 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.126502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126393 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.126502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126436 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.126502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126483 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f03d18-4be1-471b-94ae-811f8bcec9e2-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126531 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126553 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126590 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126670 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ada526-3272-44d6-a5ba-276199cb022b-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126702 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126748 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126774 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ada526-3272-44d6-a5ba-276199cb022b-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.126937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.126810 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.145760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.145163 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.145760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.145336 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.145760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.145382 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwsf\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-kube-api-access-kqwsf\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.145760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.145417 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.146131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.145950 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.187787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.184186 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.187787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.184556 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.187787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.186290 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.187787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.187434 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ada526-3272-44d6-a5ba-276199cb022b-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.191954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.191926 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.197565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.197539 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ada526-3272-44d6-a5ba-276199cb022b-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.216980 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.216480 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.238171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.238139 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwsf\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-kube-api-access-kqwsf\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-8l4s6\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.243401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.242062 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:25:58.251428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.251378 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.263869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.263829 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.263900 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shnbz\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-kube-api-access-shnbz\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.263965 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.264104 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.264160 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f03d18-4be1-471b-94ae-811f8bcec9e2-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.264220 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.264256 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.264297 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.264779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.264753 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.265047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.265026 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.265161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.265101 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.265161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.260748 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.267307 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.267283 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f03d18-4be1-471b-94ae-811f8bcec9e2-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.270410 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.270382 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.270923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.270883 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.297530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.297485 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.298707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.298668 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shnbz\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-kube-api-access-shnbz\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.349258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.348904 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:25:58.647414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.647377 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6"] Nov 03 22:25:58.652434 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:25:58.652387 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ada526_3272_44d6_a5ba_276199cb022b.slice/crio-eee3854548b60d487fe1a1a251c0b7dc3a28d5072bb3d0fa26c874459a771909 WatchSource:0}: Error finding container eee3854548b60d487fe1a1a251c0b7dc3a28d5072bb3d0fa26c874459a771909: Status 404 returned error can't find the container with id eee3854548b60d487fe1a1a251c0b7dc3a28d5072bb3d0fa26c874459a771909 Nov 03 22:25:58.659000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.658960 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:58.659108 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.659047 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:58.659177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.659116 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:58.720005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.719967 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd"] Nov 03 22:25:58.725567 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:25:58.725484 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f03d18_4be1_471b_94ae_811f8bcec9e2.slice/crio-f0eb8d6dd1d83e3a5a8f43df7fa0f15e1233bcf6f832f881deebed3e8dbee8c6 WatchSource:0}: Error finding container f0eb8d6dd1d83e3a5a8f43df7fa0f15e1233bcf6f832f881deebed3e8dbee8c6: Status 404 returned error can't find the container with id f0eb8d6dd1d83e3a5a8f43df7fa0f15e1233bcf6f832f881deebed3e8dbee8c6 Nov 03 22:25:58.731162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.731116 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:58.731274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.731225 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:58.731312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:58.731273 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:25:59.381799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:59.381755 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" event={"ID":"f0ada526-3272-44d6-a5ba-276199cb022b","Type":"ContainerStarted","Data":"2532559ed6388a5d360fbbb8dd3723f6bfaead745ad01c5ced4f0b062e60c9b7"} Nov 03 22:25:59.382299 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:59.381829 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" event={"ID":"f0ada526-3272-44d6-a5ba-276199cb022b","Type":"ContainerStarted","Data":"eee3854548b60d487fe1a1a251c0b7dc3a28d5072bb3d0fa26c874459a771909"} Nov 03 22:25:59.383622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:59.383589 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" event={"ID":"27f03d18-4be1-471b-94ae-811f8bcec9e2","Type":"ContainerStarted","Data":"79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24"} Nov 03 22:25:59.383747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:59.383647 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" event={"ID":"27f03d18-4be1-471b-94ae-811f8bcec9e2","Type":"ContainerStarted","Data":"f0eb8d6dd1d83e3a5a8f43df7fa0f15e1233bcf6f832f881deebed3e8dbee8c6"} Nov 03 22:25:59.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:59.456399 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podStartSLOduration=2.456375811 podStartE2EDuration="2.456375811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:59.438394727 +0000 UTC m=+780.867730722" watchObservedRunningTime="2026-11-03 22:25:59.456375811 +0000 UTC m=+780.885711805" Nov 03 22:25:59.502689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:25:59.502633 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podStartSLOduration=2.502618154 podStartE2EDuration="2.502618154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:59.501414504 +0000 UTC m=+780.930750498" watchObservedRunningTime="2026-11-03 22:25:59.502618154 +0000 UTC m=+780.931954147" Nov 03 22:26:00.255740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:00.252111 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:26:00.255740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:00.252295 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:26:00.255740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:00.252354 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 22:26:00.357743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:00.356970 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:26:00.357743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:00.357155 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:26:00.357743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:00.357199 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:26:01.244745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:01.244092 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:26:01.244745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:01.244175 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 22:26:01.360169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:01.360129 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:26:01.360438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:01.360405 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:26:02.255754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:02.248468 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:26:02.255754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:02.248527 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 22:26:02.323852 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:26:02.310486 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:26:02.356740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:02.352207 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:26:02.356740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:02.352292 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:26:03.245618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:03.245569 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:26:03.245959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:03.245923 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 22:26:03.352224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:03.352182 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:26:03.352957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:03.352919 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:26:03.760082 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:03.759248 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:26:03.803555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:03.803301 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:26:04.257743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:04.256802 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:26:04.257743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:04.256876 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 22:26:04.353742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:04.353267 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:26:04.353742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:04.353343 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:26:04.531663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:04.530366 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:26:05.247529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:05.247493 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:26:05.247884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:05.247850 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 22:26:05.350180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:05.350139 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:26:05.350339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:05.350207 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:26:06.248639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:06.248605 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:26:06.249133 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:06.249018 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:26:06.249650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:06.249628 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:26:06.355745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:06.355701 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:26:06.356346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:06.356312 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:26:06.357251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:06.357227 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:26:17.286545 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.286513 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9"] Nov 03 22:26:17.290591 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.290573 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.319647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.319579 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"distinct-2-w1-e19850-kserve-self-signed-certs\"" Nov 03 22:26:17.322549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.319879 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"openshift-service-ca.crt\"" Nov 03 22:26:17.322549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.320011 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"default-dockercfg-95q47\"" Nov 03 22:26:17.322549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.320555 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"kube-root-ca.crt\"" Nov 03 22:26:17.324582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.324557 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9"] Nov 03 22:26:17.421586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.421555 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-dshm\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.421743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.421626 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8q5\" (UniqueName: \"kubernetes.io/projected/ac55f107-7b16-4247-94b9-fd23281948ef-kube-api-access-5w8q5\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.421783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.421747 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-tmp-dir\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.421839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.421781 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-model-cache\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.421875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.421859 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-home\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.421919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.421907 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f107-7b16-4247-94b9-fd23281948ef-tls-certs\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.523580 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-home\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.523674 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f107-7b16-4247-94b9-fd23281948ef-tls-certs\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.523743 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-dshm\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.523818 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8q5\" (UniqueName: \"kubernetes.io/projected/ac55f107-7b16-4247-94b9-fd23281948ef-kube-api-access-5w8q5\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.523908 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-tmp-dir\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.523949 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-model-cache\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.524569 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-model-cache\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.525400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.524981 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-home\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.528339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.528305 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-tmp-dir\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.559259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.559185 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-dshm\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.568740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.567047 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f107-7b16-4247-94b9-fd23281948ef-tls-certs\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.592371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.589324 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8q5\" (UniqueName: \"kubernetes.io/projected/ac55f107-7b16-4247-94b9-fd23281948ef-kube-api-access-5w8q5\") pod \"distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.614647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.614611 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:17.964149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:17.964119 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9"] Nov 03 22:26:17.966273 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:26:17.966243 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac55f107_7b16_4247_94b9_fd23281948ef.slice/crio-967157f70402659a5ff197cbec68965d3e9ff17b4f6062feb894f7f4d6302db6 WatchSource:0}: Error finding container 967157f70402659a5ff197cbec68965d3e9ff17b4f6062feb894f7f4d6302db6: Status 404 returned error can't find the container with id 967157f70402659a5ff197cbec68965d3e9ff17b4f6062feb894f7f4d6302db6 Nov 03 22:26:18.629437 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:18.629404 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" event={"ID":"ac55f107-7b16-4247-94b9-fd23281948ef","Type":"ContainerStarted","Data":"39152919a228182e69358ae996a68dc08d303da6cdc9f803c169c55423daf23b"} Nov 03 22:26:18.629437 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:18.629442 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" event={"ID":"ac55f107-7b16-4247-94b9-fd23281948ef","Type":"ContainerStarted","Data":"967157f70402659a5ff197cbec68965d3e9ff17b4f6062feb894f7f4d6302db6"} Nov 03 22:26:18.669288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:18.669236 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podStartSLOduration=1.6692216960000001 podStartE2EDuration="1.669221696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:18.668764943 +0000 UTC m=+800.098100937" watchObservedRunningTime="2026-11-03 22:26:18.669221696 +0000 UTC m=+800.098557698" Nov 03 22:26:19.615894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:19.615865 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:27.615940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:27.615904 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:27.629188 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:27.629160 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:27.725977 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:27.725942 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:26:28.302894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.302861 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf"] Nov 03 22:26:28.307260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.307234 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.334356 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.334322 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-434ce3-openshift-default-dockercfg-4lnm5\"" Nov 03 22:26:28.340942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.340914 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf"] Nov 03 22:26:28.385812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.385774 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.385986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.385837 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.385986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.385948 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.386063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.385994 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fd4316-b749-438d-819d-1e05ef587b13-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.386099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.386081 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fd4316-b749-438d-819d-1e05ef587b13-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.386150 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.386131 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.386202 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.386162 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.386238 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.386217 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.386278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.386242 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqx7v\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-kube-api-access-dqx7v\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.491464 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.491537 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fd4316-b749-438d-819d-1e05ef587b13-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.491644 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fd4316-b749-438d-819d-1e05ef587b13-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.491704 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.491758 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.491798 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.491835 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx7v\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-kube-api-access-dqx7v\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.492000 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.492662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.492061 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.493305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.492876 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.496743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.493503 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.496743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.494099 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fd4316-b749-438d-819d-1e05ef587b13-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.496743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.494400 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.512793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.508162 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.519058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.519023 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fd4316-b749-438d-819d-1e05ef587b13-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.531095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.531057 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.552798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.552758 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqx7v\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-kube-api-access-dqx7v\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.579788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.576583 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.623118 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.623082 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:28.756398 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.756368 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w"] Nov 03 22:26:28.764010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.763984 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.790365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.790337 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w"] Nov 03 22:26:28.899122 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899087 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899142 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899219 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899242 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lfl\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-kube-api-access-59lfl\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899274 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899391 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899457 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899422 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899517 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899464 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.899615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.899587 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:28.961592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.961562 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf"] Nov 03 22:26:28.964028 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:26:28.963991 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fd4316_b749_438d_819d_1e05ef587b13.slice/crio-8c9183bbbd57b61ea33ec7958af3fabc735be52301033152e0a63c4ddac01e74 WatchSource:0}: Error finding container 8c9183bbbd57b61ea33ec7958af3fabc735be52301033152e0a63c4ddac01e74: Status 404 returned error can't find the container with id 8c9183bbbd57b61ea33ec7958af3fabc735be52301033152e0a63c4ddac01e74 Nov 03 22:26:28.966699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.966663 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:26:28.966799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.966768 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:26:28.966844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:28.966811 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:26:29.001112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001068 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001229 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001335 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001306 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001411 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001394 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001488 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001533 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001531 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001588 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001570 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001559 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.001906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.001652 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lfl\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-kube-api-access-59lfl\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.002215 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.002165 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.002462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.002425 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.002884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.002863 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.004053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.004031 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.004701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.004678 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.023930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.023907 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.024077 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.024057 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lfl\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-kube-api-access-59lfl\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.081028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.080781 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:29.422862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.422833 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w"] Nov 03 22:26:29.425050 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:26:29.425022 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b6ba1a_9a96_4ae3_af4a_413cab63c841.slice/crio-7f39e366b143cfe7f76b7eb1af554ff4362470d0ba428dba632fdde98b7318b7 WatchSource:0}: Error finding container 7f39e366b143cfe7f76b7eb1af554ff4362470d0ba428dba632fdde98b7318b7: Status 404 returned error can't find the container with id 7f39e366b143cfe7f76b7eb1af554ff4362470d0ba428dba632fdde98b7318b7 Nov 03 22:26:29.427911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.427880 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:26:29.427987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.427945 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:26:29.427987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.427972 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:26:29.771563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.770422 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" event={"ID":"c4fd4316-b749-438d-819d-1e05ef587b13","Type":"ContainerStarted","Data":"dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba"} Nov 03 22:26:29.771563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.770490 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" event={"ID":"c4fd4316-b749-438d-819d-1e05ef587b13","Type":"ContainerStarted","Data":"8c9183bbbd57b61ea33ec7958af3fabc735be52301033152e0a63c4ddac01e74"} Nov 03 22:26:29.781282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.780490 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" event={"ID":"c5b6ba1a-9a96-4ae3-af4a-413cab63c841","Type":"ContainerStarted","Data":"9444f2fc246c65a0468fc8ba88b5216c44c68a8906d07994860c6dce38994fd0"} Nov 03 22:26:29.781282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.780563 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" event={"ID":"c5b6ba1a-9a96-4ae3-af4a-413cab63c841","Type":"ContainerStarted","Data":"7f39e366b143cfe7f76b7eb1af554ff4362470d0ba428dba632fdde98b7318b7"} Nov 03 22:26:29.815760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:29.815694 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podStartSLOduration=1.815678073 podStartE2EDuration="1.815678073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:29.813673315 +0000 UTC m=+811.243009308" watchObservedRunningTime="2026-11-03 22:26:29.815678073 +0000 UTC m=+811.245014067" Nov 03 22:26:30.083601 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:30.083481 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:30.087117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:30.085783 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:30.087117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:30.085861 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:26:30.624775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:30.624736 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:30.625759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:30.625737 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:26:30.625899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:30.625878 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:26:31.084800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:31.084177 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:31.084800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:31.084245 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:26:31.624018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:31.623989 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:26:31.624224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:31.624205 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:26:32.082664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:32.082160 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:32.082664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:32.082227 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:26:32.214269 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:26:32.214233 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:26:32.624196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:32.624160 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:26:32.624380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:32.624215 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:26:33.086843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:33.085242 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:33.086843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:33.085303 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:26:33.626300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:33.626249 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:26:33.626926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:33.626875 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:26:34.092067 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:34.092025 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:34.093812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:34.093763 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:26:34.624287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:34.624250 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:26:34.624512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:34.624311 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:26:35.085798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:35.085117 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:35.085798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:35.085184 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:26:35.624011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:35.623971 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:26:35.624288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:35.624257 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:26:36.081606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:36.081517 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:36.081606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:36.081569 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:26:36.628687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:36.628655 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:36.629169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:36.629092 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:36.629852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:36.629828 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 22:26:36.692197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:36.692145 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podStartSLOduration=8.692128189 podStartE2EDuration="8.692128189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:29.870078412 +0000 UTC m=+811.299414456" watchObservedRunningTime="2026-11-03 22:26:36.692128189 +0000 UTC m=+818.121464189" Nov 03 22:26:37.086110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:37.086022 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:37.086346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:37.086324 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:26:37.087016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:26:37.087002 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:27:02.294874 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:27:02.294822 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:27:13.493416 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.493336 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s"] Nov 03 22:27:13.502990 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.502961 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.505965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.505882 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s"] Nov 03 22:27:13.527991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.527963 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"distinct-2-w6-2d619f-kserve-self-signed-certs\"" Nov 03 22:27:13.528182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.527963 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"default-dockercfg-krbfz\"" Nov 03 22:27:13.528182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.528033 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"openshift-service-ca.crt\"" Nov 03 22:27:13.528371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.528066 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"kube-root-ca.crt\"" Nov 03 22:27:13.613515 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.613486 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-home\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.613652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.613523 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30970e3-1702-45fa-bc60-bcd754afa716-tls-certs\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.613696 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.613645 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-dshm\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.613696 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.613688 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gg7c\" (UniqueName: \"kubernetes.io/projected/b30970e3-1702-45fa-bc60-bcd754afa716-kube-api-access-2gg7c\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.613837 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.613821 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-model-cache\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.613877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.613850 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-tmp-dir\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.726283 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-model-cache\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.726341 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-tmp-dir\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.726443 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-home\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.726478 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30970e3-1702-45fa-bc60-bcd754afa716-tls-certs\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.726572 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-dshm\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.726621 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gg7c\" (UniqueName: \"kubernetes.io/projected/b30970e3-1702-45fa-bc60-bcd754afa716-kube-api-access-2gg7c\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.727414 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-model-cache\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.727645 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-tmp-dir\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.732750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.727879 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-home\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.752783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.752022 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-dshm\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.764933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.759047 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30970e3-1702-45fa-bc60-bcd754afa716-tls-certs\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.764933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.764419 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gg7c\" (UniqueName: \"kubernetes.io/projected/b30970e3-1702-45fa-bc60-bcd754afa716-kube-api-access-2gg7c\") pod \"distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:13.844444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:13.844403 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:14.162589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:14.162557 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s"] Nov 03 22:27:14.167234 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:27:14.167194 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb30970e3_1702_45fa_bc60_bcd754afa716.slice/crio-da3e559cf3ceae17cfbc2d2a9ef99299b5b0d43113014ad19c438e584cbb2c03 WatchSource:0}: Error finding container da3e559cf3ceae17cfbc2d2a9ef99299b5b0d43113014ad19c438e584cbb2c03: Status 404 returned error can't find the container with id da3e559cf3ceae17cfbc2d2a9ef99299b5b0d43113014ad19c438e584cbb2c03 Nov 03 22:27:14.395751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:14.391226 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" event={"ID":"b30970e3-1702-45fa-bc60-bcd754afa716","Type":"ContainerStarted","Data":"60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1"} Nov 03 22:27:14.395751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:14.391278 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" event={"ID":"b30970e3-1702-45fa-bc60-bcd754afa716","Type":"ContainerStarted","Data":"da3e559cf3ceae17cfbc2d2a9ef99299b5b0d43113014ad19c438e584cbb2c03"} Nov 03 22:27:14.449827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:14.444986 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podStartSLOduration=1.444965438 podStartE2EDuration="1.444965438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:14.441418079 +0000 UTC m=+855.870754072" watchObservedRunningTime="2026-11-03 22:27:14.444965438 +0000 UTC m=+855.874301430" Nov 03 22:27:14.845544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:14.845506 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:23.845500 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:23.845464 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:23.858808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:23.858782 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:24.498219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:24.498193 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:27:28.226784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.226709 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6"] Nov 03 22:27:28.243843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.242180 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.342821 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6"] Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.345813 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8965\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-kube-api-access-v8965\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.345875 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.345964 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.346017 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.346073 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.346126 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.346164 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.346252 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.346775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.346328 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.447573 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447517 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.447818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447622 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.447818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447684 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8965\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-kube-api-access-v8965\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.447818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447748 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.448084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447835 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.448084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447886 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.448084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447941 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.448084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.447999 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.448084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.448041 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.449046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.449017 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.449895 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.449865 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.450161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.450133 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.451832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.451806 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.452089 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.452069 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.456048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.456001 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.461586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.461553 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.476014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.475945 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.477275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.477201 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8965\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-kube-api-access-v8965\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-jzgb6\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:28.646740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:28.641254 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:29.812822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:29.812124 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6"] Nov 03 22:27:29.812822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:29.812281 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:29.845460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:29.845401 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:29.845790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:29.845757 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:30.012628 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.012588 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" event={"ID":"fb52bd42-200f-45fb-a87e-fd9b94b173b4","Type":"ContainerStarted","Data":"ef7c44268e918e1efcf652b018feeb89e85fd0a536345cafdfb7f85bea77fa1e"} Nov 03 22:27:30.362944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.362855 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb"] Nov 03 22:27:30.409880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.409850 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb"] Nov 03 22:27:30.410224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.410202 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.463890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.429155 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"openshift-service-ca.crt\"" Nov 03 22:27:30.496490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.464416 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"kube-root-ca.crt\"" Nov 03 22:27:30.496490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.464518 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"distinct-2-w5-434ce3-kserve-self-signed-certs\"" Nov 03 22:27:30.496490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.464565 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"default-dockercfg-qg8d5\"" Nov 03 22:27:30.555741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.532873 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-tmp-dir\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.556078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.556049 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0a5b4-30e8-440b-8a10-832f72c5e885-tls-certs\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.556250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.556230 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkb26\" (UniqueName: \"kubernetes.io/projected/9de0a5b4-30e8-440b-8a10-832f72c5e885-kube-api-access-fkb26\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.567796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.556521 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-home\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.609053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.609012 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-dshm\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.609455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.609413 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-model-cache\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.717789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.716809 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-home\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.717789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.716940 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-dshm\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.734291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.734259 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-home\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.737035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.735426 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-model-cache\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.737035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.735590 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-tmp-dir\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.737035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.735644 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0a5b4-30e8-440b-8a10-832f72c5e885-tls-certs\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.737035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.735687 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkb26\" (UniqueName: \"kubernetes.io/projected/9de0a5b4-30e8-440b-8a10-832f72c5e885-kube-api-access-fkb26\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.737035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.736518 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-tmp-dir\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.795748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.748190 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-model-cache\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.795748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.761563 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-dshm\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.804361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.803535 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkb26\" (UniqueName: \"kubernetes.io/projected/9de0a5b4-30e8-440b-8a10-832f72c5e885-kube-api-access-fkb26\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.854775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.853121 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0a5b4-30e8-440b-8a10-832f72c5e885-tls-certs\") pod \"distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:30.937778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:30.937702 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:31.025901 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:31.025417 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" event={"ID":"fb52bd42-200f-45fb-a87e-fd9b94b173b4","Type":"ContainerStarted","Data":"62654581063edafb736044850dab8447c1996bcb9e53832c56693a92b411762d"} Nov 03 22:27:31.072498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:31.072434 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podStartSLOduration=3.072413349 podStartE2EDuration="3.072413349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:31.069120208 +0000 UTC m=+872.498456277" watchObservedRunningTime="2026-11-03 22:27:31.072413349 +0000 UTC m=+872.501749356" Nov 03 22:27:31.253805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:31.253691 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb"] Nov 03 22:27:31.257179 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:27:31.257146 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9de0a5b4_30e8_440b_8a10_832f72c5e885.slice/crio-7e16f5c73c1beecb5ca3f6aa4323b5b3a74fd3ea9a2435cbd082be69b799c4cf WatchSource:0}: Error finding container 7e16f5c73c1beecb5ca3f6aa4323b5b3a74fd3ea9a2435cbd082be69b799c4cf: Status 404 returned error can't find the container with id 7e16f5c73c1beecb5ca3f6aa4323b5b3a74fd3ea9a2435cbd082be69b799c4cf Nov 03 22:27:31.649586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:31.649551 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:31.649970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:31.649948 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:27:31.650117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:31.650094 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:27:32.038038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:32.037945 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" event={"ID":"9de0a5b4-30e8-440b-8a10-832f72c5e885","Type":"ContainerStarted","Data":"5135fe300efd3fee97b61c94442a56afd296d08a3587889dacee5dff582ea80b"} Nov 03 22:27:32.038595 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:32.038566 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" event={"ID":"9de0a5b4-30e8-440b-8a10-832f72c5e885","Type":"ContainerStarted","Data":"7e16f5c73c1beecb5ca3f6aa4323b5b3a74fd3ea9a2435cbd082be69b799c4cf"} Nov 03 22:27:32.081285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:32.081227 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podStartSLOduration=2.081211983 podStartE2EDuration="2.081211983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:32.079407849 +0000 UTC m=+873.508743845" watchObservedRunningTime="2026-11-03 22:27:32.081211983 +0000 UTC m=+873.510547978" Nov 03 22:27:32.284741 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:27:32.283895 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:27:32.645563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:32.645127 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:27:32.645563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:32.645191 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:27:32.942805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:32.941815 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:33.646168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:33.646127 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:27:33.646816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:33.646773 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:27:34.650169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:34.650132 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:27:34.650815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:34.650783 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:27:35.647150 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:35.647110 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:27:35.647442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:35.647412 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:27:36.648142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:36.648102 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:27:36.648797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:36.648761 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:27:37.644936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:37.644897 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:27:37.645148 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:37.644966 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:27:38.649542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:38.649509 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:38.650008 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:38.649986 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:38.650835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:38.650812 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:27:40.937949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:40.937917 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:40.951042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:40.951008 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:41.231149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:41.231068 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:27:43.148759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.148703 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h"] Nov 03 22:27:43.153520 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.153491 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.184687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.184649 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h"] Nov 03 22:27:43.277526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.277481 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.277782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.277616 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.277782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.277647 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trs76\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-kube-api-access-trs76\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.277782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.277757 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.277929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.277805 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.277929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.277914 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.278005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.277960 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.278068 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.278055 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.278104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.278079 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379143 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379080 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379143 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379146 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379212 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379286 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379311 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trs76\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-kube-api-access-trs76\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379343 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379389 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379485 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379545 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379587 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.379732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379621 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.380000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.379946 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.380142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.380078 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.380261 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.380242 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.382183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.382152 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.382299 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.382159 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.402336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.402248 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.402336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.402276 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trs76\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-kube-api-access-trs76\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.473502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.473462 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:43.809931 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.809905 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h"] Nov 03 22:27:43.812185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:27:43.812146 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60a3457_94ee_4bb6_a9f8_7d32d0b63378.slice/crio-4e242f65a1cff7f2facf13f6380586cf63fc5bac97d2a73fa03cfb9cbfa9e4b1 WatchSource:0}: Error finding container 4e242f65a1cff7f2facf13f6380586cf63fc5bac97d2a73fa03cfb9cbfa9e4b1: Status 404 returned error can't find the container with id 4e242f65a1cff7f2facf13f6380586cf63fc5bac97d2a73fa03cfb9cbfa9e4b1 Nov 03 22:27:43.815343 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.815307 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:43.815455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.815391 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:43.815455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.815435 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:43.832089 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.832059 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d"] Nov 03 22:27:43.838464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.838438 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.858708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.857885 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d"] Nov 03 22:27:43.987801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.987764 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.987946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.987915 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.988025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.987957 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791a532-f4ab-40e4-9f0c-39631d879947-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.988025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.987991 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.988177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.988154 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791a532-f4ab-40e4-9f0c-39631d879947-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.988405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.988289 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.988405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.988363 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.988405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.988392 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:43.988617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:43.988473 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7vv\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-kube-api-access-rm7vv\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.089500 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.089460 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.089690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.089507 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791a532-f4ab-40e4-9f0c-39631d879947-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.089690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.089532 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.089690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.089586 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791a532-f4ab-40e4-9f0c-39631d879947-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.089914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.089695 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090029 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090078 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090117 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090207 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7vv\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-kube-api-access-rm7vv\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090257 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090278 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090435 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791a532-f4ab-40e4-9f0c-39631d879947-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090519 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.090612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.090594 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.092148 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.092125 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.093245 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.093224 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791a532-f4ab-40e4-9f0c-39631d879947-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.112236 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.112198 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.112418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.112364 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7vv\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-kube-api-access-rm7vv\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.192816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.192020 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:44.260878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.260831 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" event={"ID":"d60a3457-94ee-4bb6-a9f8-7d32d0b63378","Type":"ContainerStarted","Data":"9b1718bfeb88f6a969ac2cbb857b9c531b82d0d43e67e83eaffce908a66e8d88"} Nov 03 22:27:44.261056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.260941 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" event={"ID":"d60a3457-94ee-4bb6-a9f8-7d32d0b63378","Type":"ContainerStarted","Data":"4e242f65a1cff7f2facf13f6380586cf63fc5bac97d2a73fa03cfb9cbfa9e4b1"} Nov 03 22:27:44.474359 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.474256 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:44.476052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.476021 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:44.476196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.476071 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:27:44.575323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.575265 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podStartSLOduration=1.5752453100000001 podStartE2EDuration="1.57524531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:44.306143866 +0000 UTC m=+885.735479861" watchObservedRunningTime="2026-11-03 22:27:44.57524531 +0000 UTC m=+886.004581307" Nov 03 22:27:44.578389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.577778 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d"] Nov 03 22:27:44.578389 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:27:44.578169 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6791a532_f4ab_40e4_9f0c_39631d879947.slice/crio-33b4ccb969495018643b5d5f2624ecb7646d274a9950a96f2f6472db8cf546c8 WatchSource:0}: Error finding container 33b4ccb969495018643b5d5f2624ecb7646d274a9950a96f2f6472db8cf546c8: Status 404 returned error can't find the container with id 33b4ccb969495018643b5d5f2624ecb7646d274a9950a96f2f6472db8cf546c8 Nov 03 22:27:44.581279 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.581242 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:44.581387 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.581314 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:44.581387 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:44.581343 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:27:45.293039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:45.292994 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" event={"ID":"6791a532-f4ab-40e4-9f0c-39631d879947","Type":"ContainerStarted","Data":"2c0bd70fd887127f8d8cca306a436b1df10eab1819d6a01dde8a377b46c74e47"} Nov 03 22:27:45.293433 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:45.293062 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" event={"ID":"6791a532-f4ab-40e4-9f0c-39631d879947","Type":"ContainerStarted","Data":"33b4ccb969495018643b5d5f2624ecb7646d274a9950a96f2f6472db8cf546c8"} Nov 03 22:27:45.345295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:45.345225 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podStartSLOduration=2.345199163 podStartE2EDuration="2.345199163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:45.341190857 +0000 UTC m=+886.770526855" watchObservedRunningTime="2026-11-03 22:27:45.345199163 +0000 UTC m=+886.774535158" Nov 03 22:27:45.474166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:45.474128 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:45.474450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:45.474419 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:27:46.196742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:46.195736 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:46.196742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:46.195830 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:27:46.196742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:46.196200 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:46.478802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:46.476370 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:46.478802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:46.476436 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:27:47.197742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:47.197232 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:47.197742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:47.197303 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:27:47.481528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:47.481432 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:47.482139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:47.482104 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:27:48.194666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:48.194627 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:48.194884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:48.194686 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:27:48.478800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:48.478159 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:48.478800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:48.478224 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:27:49.193896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:49.193854 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:49.194301 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:49.193913 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:27:49.483798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:49.477178 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:49.483798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:49.477242 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:27:50.194326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:50.194279 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:50.194870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:50.194344 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:27:50.474801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:50.474206 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:27:50.474801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:50.474267 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:27:51.194155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:51.194120 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:27:51.194360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:51.194188 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:27:51.823326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:51.823291 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:51.824459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:51.824422 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:51.826399 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:51.826384 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:27:52.353902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:52.353867 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:52.355393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:52.355369 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:52.356230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:52.356212 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:27:58.486542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.480852 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6"] Nov 03 22:27:58.492959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.492542 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.501564 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.501532 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6"] Nov 03 22:27:58.560578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.560537 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6fed8b-717b-4057-849a-69a007087715-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.560962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.560938 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.561210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.561189 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.561451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.561431 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.561675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.561657 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.562047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.562023 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.562371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.562340 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.562607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.562574 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbwj\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-kube-api-access-pdbwj\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.562900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.562863 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6fed8b-717b-4057-849a-69a007087715-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.684239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.684198 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.691846 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.691800 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.692217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.692182 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.692429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.692398 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.692631 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.692615 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.692865 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.692849 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbwj\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-kube-api-access-pdbwj\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.693111 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.693096 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6fed8b-717b-4057-849a-69a007087715-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.693298 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.693281 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6fed8b-717b-4057-849a-69a007087715-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.693440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.693427 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.693617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.693600 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.694312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.694291 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.708763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.694781 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.710566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.710539 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6fed8b-717b-4057-849a-69a007087715-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.833800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.823387 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.847782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.835783 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.884782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.855605 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.884782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.872342 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6fed8b-717b-4057-849a-69a007087715-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:58.884782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:58.875143 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbwj\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-kube-api-access-pdbwj\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:59.134306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:59.134276 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:27:59.289061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:59.289022 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:27:59.353288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:27:59.353217 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:28:00.241758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:00.228134 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.518741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:00.506706 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.590823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:00.583872 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:00.590823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:00.583969 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:00.590823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:00.584015 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:00.743894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:00.743836 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:28:01.265741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:01.197744 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6"] Nov 03 22:28:01.326117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:01.326077 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" event={"ID":"af6fed8b-717b-4057-849a-69a007087715","Type":"ContainerStarted","Data":"d192b2ed33617c1cb2a38315f8b52c641e046c3a4b28eb61aa2415016aa247e8"} Nov 03 22:28:01.457016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:01.456963 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:01.555795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:01.551278 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:01.555795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:01.551872 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:28:02.048358 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:02.041899 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Nov 03 22:28:02.289941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:02.289898 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:28:02.422954 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:02.422912 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:28:02.475155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:02.475093 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:02.479765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:02.477599 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:28:02.506333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:02.506303 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:28:02.580769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:02.580665 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:28:03.174407 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:03.174363 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" event={"ID":"af6fed8b-717b-4057-849a-69a007087715","Type":"ContainerStarted","Data":"ca2addaadeab28f9d6cb6ae9b466f2d203c20fe6444d67a173f0aff8932b14f1"} Nov 03 22:28:03.218426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:03.218369 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podStartSLOduration=5.218354519 podStartE2EDuration="5.218354519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:03.21682932 +0000 UTC m=+904.646165319" watchObservedRunningTime="2026-11-03 22:28:03.218354519 +0000 UTC m=+904.647690512" Nov 03 22:28:04.143754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:04.142368 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:28:04.143754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:04.142577 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:04.143754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:04.142627 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:28:05.137200 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:05.137146 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:05.137502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:05.137471 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:28:06.138180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:06.138139 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:06.140636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:06.140605 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:28:07.137219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:07.137175 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:07.137566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:07.137530 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:28:08.142753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:08.142179 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:08.142753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:08.142245 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:28:09.142784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:09.142408 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:09.142784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:09.142472 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:28:10.143007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:10.142969 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:28:10.144362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:10.144340 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:28:10.145222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:10.145203 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:28:13.348401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.348367 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk"] Nov 03 22:28:13.353894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.353873 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.377767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.377736 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk"] Nov 03 22:28:13.458825 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.458779 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.458825 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.458821 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce339ce7-c721-474f-9f19-fabec26bffef-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.459123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.458938 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.459123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.459062 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klg4w\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-kube-api-access-klg4w\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.459123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.459120 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.459329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.459182 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce339ce7-c721-474f-9f19-fabec26bffef-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.459329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.459215 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.459329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.459259 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.459329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.459279 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.560838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.560799 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.560938 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klg4w\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-kube-api-access-klg4w\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.560982 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.560999 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce339ce7-c721-474f-9f19-fabec26bffef-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561039 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561171 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561206 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561338 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561366 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce339ce7-c721-474f-9f19-fabec26bffef-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561382 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561506 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561544 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.561663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.561589 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.562019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.562002 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce339ce7-c721-474f-9f19-fabec26bffef-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.563811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.563789 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.564170 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.564152 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce339ce7-c721-474f-9f19-fabec26bffef-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.593969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.593908 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klg4w\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-kube-api-access-klg4w\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.594335 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.594313 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-mn9sk\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:13.679534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.679442 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:14.000823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:13.998175 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs"] Nov 03 22:28:14.004357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.004327 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.028354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.025558 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs"] Nov 03 22:28:14.066106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.066070 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk"] Nov 03 22:28:14.067063 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:14.067026 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce339ce7_c721_474f_9f19_fabec26bffef.slice/crio-ffe4d62a753f990e062d009376ddc9a9b4f585691cbd42d199bd8adf25bd5e3e WatchSource:0}: Error finding container ffe4d62a753f990e062d009376ddc9a9b4f585691cbd42d199bd8adf25bd5e3e: Status 404 returned error can't find the container with id ffe4d62a753f990e062d009376ddc9a9b4f585691cbd42d199bd8adf25bd5e3e Nov 03 22:28:14.070509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.070469 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:14.070607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.070555 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:14.070607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.070597 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:14.111847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.111816 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2671461-d398-44a6-a9c9-71917ce1d589-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.111913 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.111968 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2671461-d398-44a6-a9c9-71917ce1d589-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.112143 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.112181 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.112271 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112341 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.112315 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whfv9\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-kube-api-access-whfv9\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.112408 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.112530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.112440 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.213680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.213642 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.213680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.213692 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.213945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.213744 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.213945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.213786 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfv9\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-kube-api-access-whfv9\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.213945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.213849 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.213945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.213889 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.214168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.213979 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2671461-d398-44a6-a9c9-71917ce1d589-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.214168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.214033 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.214168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.214114 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2671461-d398-44a6-a9c9-71917ce1d589-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.214319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.214187 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.214875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.214418 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.214875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.214762 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.214875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.214841 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.215085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.214968 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2671461-d398-44a6-a9c9-71917ce1d589-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.217314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.217282 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.217412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.217390 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2671461-d398-44a6-a9c9-71917ce1d589-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.236577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.236544 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.236841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.236801 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfv9\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-kube-api-access-whfv9\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.295791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.295669 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" event={"ID":"ce339ce7-c721-474f-9f19-fabec26bffef","Type":"ContainerStarted","Data":"96740c400e3fb90614082db87821e7c770ef1efcb53c0196728840e50ebf0e6c"} Nov 03 22:28:14.296052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.296029 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" event={"ID":"ce339ce7-c721-474f-9f19-fabec26bffef","Type":"ContainerStarted","Data":"ffe4d62a753f990e062d009376ddc9a9b4f585691cbd42d199bd8adf25bd5e3e"} Nov 03 22:28:14.330544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.330511 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:14.340585 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.340535 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podStartSLOduration=1.340518764 podStartE2EDuration="1.340518764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:14.338288174 +0000 UTC m=+915.767624179" watchObservedRunningTime="2026-11-03 22:28:14.340518764 +0000 UTC m=+915.769854756" Nov 03 22:28:14.673652 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:14.673618 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2671461_d398_44a6_a9c9_71917ce1d589.slice/crio-7b6f34d4645f34abe5bbb35d3d42d2353c42808ed4092a82b2ed19cbeb764be5 WatchSource:0}: Error finding container 7b6f34d4645f34abe5bbb35d3d42d2353c42808ed4092a82b2ed19cbeb764be5: Status 404 returned error can't find the container with id 7b6f34d4645f34abe5bbb35d3d42d2353c42808ed4092a82b2ed19cbeb764be5 Nov 03 22:28:14.676531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.676497 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:14.676627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.676567 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:14.676627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.676604 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:14.679212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.679073 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs"] Nov 03 22:28:14.680059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:14.680036 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:15.305077 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:15.305039 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" event={"ID":"f2671461-d398-44a6-a9c9-71917ce1d589","Type":"ContainerStarted","Data":"08e68b193f8db9b58dc28725062d445fd4584f61a4d4725c8a966b1217a6280f"} Nov 03 22:28:15.305297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:15.305103 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" event={"ID":"f2671461-d398-44a6-a9c9-71917ce1d589","Type":"ContainerStarted","Data":"7b6f34d4645f34abe5bbb35d3d42d2353c42808ed4092a82b2ed19cbeb764be5"} Nov 03 22:28:15.330818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:15.330782 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:15.332432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:15.332399 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:28:15.332693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:15.332659 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:28:15.684342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:15.684294 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:28:15.684981 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:15.684946 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:28:16.331894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:16.331858 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:28:16.332076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:16.331913 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:28:16.682205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:16.682166 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:28:16.683412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:16.683379 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:28:17.333740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:17.333215 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:28:17.333740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:17.333287 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:28:17.697386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:17.691290 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:28:17.697386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:17.691358 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:28:18.331428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:18.331390 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:28:18.331619 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:18.331445 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:28:18.683754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:18.682179 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:28:18.683754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:18.682243 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:28:19.331647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:19.331610 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:28:19.331882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:19.331664 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:28:19.690744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:19.681254 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:28:19.690744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:19.681321 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:28:20.331163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:20.331128 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:28:20.331370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:20.331184 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:28:20.681391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:20.681345 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:28:20.681584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:20.681411 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:28:21.331366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:21.331330 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:28:21.331819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:21.331392 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:28:21.893999 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:21.893959 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:21.894972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:21.894938 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:21.896481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:21.896458 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:28:21.960227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:21.960163 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podStartSLOduration=8.960148042 podStartE2EDuration="8.960148042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:15.371533275 +0000 UTC m=+916.800869267" watchObservedRunningTime="2026-11-03 22:28:21.960148042 +0000 UTC m=+923.389484035" Nov 03 22:28:22.548079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:22.548050 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:22.549729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:22.549641 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:22.550797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:22.550776 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:28:25.672901 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:25.672866 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-049167"] Nov 03 22:28:25.677367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:25.677345 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:25.691884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:25.691857 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-khmwt\"" Nov 03 22:28:25.693157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:25.693133 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-049167"] Nov 03 22:28:25.811862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:25.811828 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfd5p\" (UniqueName: \"kubernetes.io/projected/28101e0f-0721-4f73-b446-6de97f0b7997-kube-api-access-nfd5p\") pod \"test-curl-7257-049167\" (UID: \"28101e0f-0721-4f73-b446-6de97f0b7997\") " pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:25.913180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:25.913145 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfd5p\" (UniqueName: \"kubernetes.io/projected/28101e0f-0721-4f73-b446-6de97f0b7997-kube-api-access-nfd5p\") pod \"test-curl-7257-049167\" (UID: \"28101e0f-0721-4f73-b446-6de97f0b7997\") " pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:25.936024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:25.935944 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfd5p\" (UniqueName: \"kubernetes.io/projected/28101e0f-0721-4f73-b446-6de97f0b7997-kube-api-access-nfd5p\") pod \"test-curl-7257-049167\" (UID: \"28101e0f-0721-4f73-b446-6de97f0b7997\") " pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:26.006330 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:26.006010 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:26.367117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:26.367081 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-049167"] Nov 03 22:28:26.369109 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:26.369074 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28101e0f_0721_4f73_b446_6de97f0b7997.slice/crio-d93db423506ec5100d57fe862e6c8487331e143774ae344968400c6b19092e2f WatchSource:0}: Error finding container d93db423506ec5100d57fe862e6c8487331e143774ae344968400c6b19092e2f: Status 404 returned error can't find the container with id d93db423506ec5100d57fe862e6c8487331e143774ae344968400c6b19092e2f Nov 03 22:28:26.477371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:26.477333 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-049167" event={"ID":"28101e0f-0721-4f73-b446-6de97f0b7997","Type":"ContainerStarted","Data":"d93db423506ec5100d57fe862e6c8487331e143774ae344968400c6b19092e2f"} Nov 03 22:28:28.705737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.705040 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7"] Nov 03 22:28:28.713735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.712785 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793419 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793470 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793566 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793624 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793660 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793697 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d419320-4e51-4e47-bb17-6a669692cf74-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793759 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d419320-4e51-4e47-bb17-6a669692cf74-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793804 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.806749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.793873 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdfv\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-kube-api-access-5pdfv\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.828735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.826702 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7"] Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.907936 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908020 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908095 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d419320-4e51-4e47-bb17-6a669692cf74-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908166 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d419320-4e51-4e47-bb17-6a669692cf74-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908216 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908305 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdfv\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-kube-api-access-5pdfv\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908543 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.908741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908578 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.909298 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.908782 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.968796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.968565 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.984742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.969051 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:28.984742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:28.969653 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d419320-4e51-4e47-bb17-6a669692cf74-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:29.031741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:29.030759 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:29.031741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:29.031396 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:29.040406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:29.033963 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:29.059773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:29.048667 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d419320-4e51-4e47-bb17-6a669692cf74-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:29.149744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:29.096689 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdfv\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-kube-api-access-5pdfv\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:29.149744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:29.104317 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:29.398746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:29.391988 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:30.749815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:30.745651 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:28:30.770817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:30.753835 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.135325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.135286 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7"] Nov 03 22:28:31.248747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.245915 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:28:31.345741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.336933 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:31.345741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.337037 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:31.345741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.337084 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:31.796740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.771657 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.796740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.772398 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:28:31.810678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.810262 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:28:31.877754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.869830 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:28:31.928758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.922891 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:28:31.972742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:31.965184 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.282351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:32.254406 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.836997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:32.791123 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:28:32.836997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:32.824797 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:33.177896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:33.155983 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:28:33.519742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:33.496154 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:28:33.710287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:33.687020 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:28:33.731643 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:33.731601 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:28:33.852747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:33.839815 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:34.057694 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:34.057599 4625 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.205s" Nov 03 22:28:34.057985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.057965 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:28:34.195751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.176894 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:34.264774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.222841 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:28:34.264774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.264618 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:34.325385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.324829 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" event={"ID":"1d419320-4e51-4e47-bb17-6a669692cf74","Type":"ContainerStarted","Data":"88bd031c33fb560b56b60c159d68cba521a1d5071e236e5bde07ac9fee8db89e"} Nov 03 22:28:34.707744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.688223 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:34.756394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.756317 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:34.767275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.767233 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:34.878034 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:34.877971 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:35.015252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:35.014846 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:35.271926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:35.271815 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:35.436794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:35.435554 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" event={"ID":"1d419320-4e51-4e47-bb17-6a669692cf74","Type":"ContainerStarted","Data":"940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc"} Nov 03 22:28:35.523801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:35.520416 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:28:35.527747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:35.526591 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podStartSLOduration=7.526573164 podStartE2EDuration="7.526573164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:35.499021906 +0000 UTC m=+936.928357900" watchObservedRunningTime="2026-11-03 22:28:35.526573164 +0000 UTC m=+936.955909159" Nov 03 22:28:35.808793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:35.690772 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:35.913006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:35.892611 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:28:36.181550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:36.181094 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:28:36.221740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:36.221090 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:28:36.534761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:36.534647 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:28:36.556351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:36.540568 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:36.891788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:36.891751 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:28:36.891999 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:36.891834 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 22:28:37.179661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.179574 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm"] Nov 03 22:28:37.184843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.184819 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.200243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.200212 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm"] Nov 03 22:28:37.205250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.205227 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:28:37.205386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.205305 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:37.300558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.300519 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.301012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.300986 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mcv\" (UniqueName: \"kubernetes.io/projected/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-kube-api-access-69mcv\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.301147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.301027 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.393012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.392971 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:28:37.393198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.393035 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 22:28:37.402903 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.402867 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mcv\" (UniqueName: \"kubernetes.io/projected/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-kube-api-access-69mcv\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.403056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.402917 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.403117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.403078 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.404086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.404056 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.405427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.405408 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.425574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.425544 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mcv\" (UniqueName: \"kubernetes.io/projected/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-kube-api-access-69mcv\") pod \"maas-api-revoke-sub-831ef85e-dhcqm\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.504438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.504359 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:37.983188 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:37.982594 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm"] Nov 03 22:28:38.392962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:38.392931 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:28:38.393363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:38.392981 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 22:28:39.007186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:39.007146 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" event={"ID":"1c19a6f7-87bc-45e7-9e86-62b09ab69bee","Type":"ContainerStarted","Data":"c9a20b7cf2216fe3ef0ede86299f16ee6f4ac0a503c02c416036d41af22730e7"} Nov 03 22:28:39.007372 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:39.007207 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" event={"ID":"1c19a6f7-87bc-45e7-9e86-62b09ab69bee","Type":"ContainerStarted","Data":"46066b520ae05e08cec96158bbf666fa8ca76fd5d08b14d070b956208c3c848a"} Nov 03 22:28:39.398849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:39.397111 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:28:39.398849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:39.397173 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 22:28:40.016010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:40.015971 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-049167" event={"ID":"28101e0f-0721-4f73-b446-6de97f0b7997","Type":"ContainerStarted","Data":"a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1"} Nov 03 22:28:40.026599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:40.026559 4625 generic.go:356] "Generic (PLEG): container finished" podID="1c19a6f7-87bc-45e7-9e86-62b09ab69bee" containerID="c9a20b7cf2216fe3ef0ede86299f16ee6f4ac0a503c02c416036d41af22730e7" exitCode=0 Nov 03 22:28:40.026815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:40.026638 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" event={"ID":"1c19a6f7-87bc-45e7-9e86-62b09ab69bee","Type":"ContainerDied","Data":"c9a20b7cf2216fe3ef0ede86299f16ee6f4ac0a503c02c416036d41af22730e7"} Nov 03 22:28:40.070342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:40.070282 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" podStartSLOduration=3.070265171 podStartE2EDuration="3.070265171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:39.069151189 +0000 UTC m=+940.498487182" watchObservedRunningTime="2026-11-03 22:28:40.070265171 +0000 UTC m=+941.499601164" Nov 03 22:28:40.071465 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:40.071425 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-049167" podStartSLOduration=1.929677153 podStartE2EDuration="15.071414024s" totalImagesPullingTime="13.141736871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:40.066986867 +0000 UTC m=+941.496322860" watchObservedRunningTime="2026-11-03 22:28:40.071414024 +0000 UTC m=+941.500750017" Nov 03 22:28:40.392972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:40.392930 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:28:40.393139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:40.392985 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 22:28:41.112026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.111915 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-049167"] Nov 03 22:28:41.112929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.112902 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-049167"] Nov 03 22:28:41.207547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.207511 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-044416"] Nov 03 22:28:41.208193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.208177 4625 cpu_manager.go:414] "containerMap: removing container" podUID="28101e0f-0721-4f73-b446-6de97f0b7997" containerName="test-curl-7257-049167" Nov 03 22:28:41.208257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.208193 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28101e0f-0721-4f73-b446-6de97f0b7997" containerName="test-curl-7257-049167" Nov 03 22:28:41.208293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.208282 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="28101e0f-0721-4f73-b446-6de97f0b7997" containerName="test-curl-7257-049167" Nov 03 22:28:41.211820 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.211801 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:41.228119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.228030 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-044416"] Nov 03 22:28:41.282149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.282080 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2dl\" (UniqueName: \"kubernetes.io/projected/6af996a6-0fda-45ff-aed7-13a9873fdbe5-kube-api-access-cb2dl\") pod \"test-curl-7257-044416\" (UID: \"6af996a6-0fda-45ff-aed7-13a9873fdbe5\") " pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:41.383790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.383573 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2dl\" (UniqueName: \"kubernetes.io/projected/6af996a6-0fda-45ff-aed7-13a9873fdbe5-kube-api-access-cb2dl\") pod \"test-curl-7257-044416\" (UID: \"6af996a6-0fda-45ff-aed7-13a9873fdbe5\") " pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:41.393009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.392921 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:28:41.393009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.392965 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 22:28:41.405663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.405638 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2dl\" (UniqueName: \"kubernetes.io/projected/6af996a6-0fda-45ff-aed7-13a9873fdbe5-kube-api-access-cb2dl\") pod \"test-curl-7257-044416\" (UID: \"6af996a6-0fda-45ff-aed7-13a9873fdbe5\") " pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:41.425079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.425045 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:41.485793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.484670 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-kube-api-access-69mcv\" (UniqueName: \"kubernetes.io/projected/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-kube-api-access-69mcv\") pod \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " Nov 03 22:28:41.485793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.484817 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-openshift-ca\") pod \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " Nov 03 22:28:41.485793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.484928 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-xks-ca\") pod \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\" (UID: \"1c19a6f7-87bc-45e7-9e86-62b09ab69bee\") " Nov 03 22:28:41.486135 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.486087 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-openshift-ca" pod "1c19a6f7-87bc-45e7-9e86-62b09ab69bee" (UID: "1c19a6f7-87bc-45e7-9e86-62b09ab69bee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:41.493333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.493289 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-xks-ca" pod "1c19a6f7-87bc-45e7-9e86-62b09ab69bee" (UID: "1c19a6f7-87bc-45e7-9e86-62b09ab69bee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:41.494193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.494156 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-kube-api-access-69mcv" pod "1c19a6f7-87bc-45e7-9e86-62b09ab69bee" (UID: "1c19a6f7-87bc-45e7-9e86-62b09ab69bee"). InnerVolumeSpecName "kube-api-access-69mcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:41.525926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.525901 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:41.586226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.586198 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:28:41.586226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.586229 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:28:41.586226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.586238 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69mcv\" (UniqueName: \"kubernetes.io/projected/1c19a6f7-87bc-45e7-9e86-62b09ab69bee-kube-api-access-69mcv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:28:41.875224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:41.875199 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-044416"] Nov 03 22:28:41.878322 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:41.878294 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6af996a6_0fda_45ff_aed7_13a9873fdbe5.slice/crio-38b74742c3594df0077fec73cab5e87192fc80722b3c7311ece568d6bb0dd809 WatchSource:0}: Error finding container 38b74742c3594df0077fec73cab5e87192fc80722b3c7311ece568d6bb0dd809: Status 404 returned error can't find the container with id 38b74742c3594df0077fec73cab5e87192fc80722b3c7311ece568d6bb0dd809 Nov 03 22:28:42.127843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.127693 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-044416" event={"ID":"6af996a6-0fda-45ff-aed7-13a9873fdbe5","Type":"ContainerStarted","Data":"aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4"} Nov 03 22:28:42.127843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.127796 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-044416" event={"ID":"6af996a6-0fda-45ff-aed7-13a9873fdbe5","Type":"ContainerStarted","Data":"38b74742c3594df0077fec73cab5e87192fc80722b3c7311ece568d6bb0dd809"} Nov 03 22:28:42.129252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.129232 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" event={"ID":"1c19a6f7-87bc-45e7-9e86-62b09ab69bee","Type":"ContainerDied","Data":"46066b520ae05e08cec96158bbf666fa8ca76fd5d08b14d070b956208c3c848a"} Nov 03 22:28:42.129319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.129251 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm" Nov 03 22:28:42.129319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.129278 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46066b520ae05e08cec96158bbf666fa8ca76fd5d08b14d070b956208c3c848a" Nov 03 22:28:42.129319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.129298 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-049167" podUID="28101e0f-0721-4f73-b446-6de97f0b7997" containerName="test-curl-7257-049167" containerID="cri-o://a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1" gracePeriod=2 Nov 03 22:28:42.170084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.170023 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-044416" podStartSLOduration=1.170003062 podStartE2EDuration="1.170003062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:42.168375393 +0000 UTC m=+943.597711387" watchObservedRunningTime="2026-11-03 22:28:42.170003062 +0000 UTC m=+943.599339056" Nov 03 22:28:42.182899 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:42.181054 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-049167\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="28101e0f-0721-4f73-b446-6de97f0b7997" pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:42.393000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.392919 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 22:28:42.393000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:42.392984 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 22:28:43.066944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.066900 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-044416"] Nov 03 22:28:43.074553 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.074522 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-044416"] Nov 03 22:28:43.584527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.584488 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l"] Nov 03 22:28:43.585578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.585552 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" containerName="test-curl-7257-044416" Nov 03 22:28:43.585578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.585575 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" containerName="test-curl-7257-044416" Nov 03 22:28:43.585831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.585624 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1c19a6f7-87bc-45e7-9e86-62b09ab69bee" containerName="revoke-keys" Nov 03 22:28:43.585831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.585630 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c19a6f7-87bc-45e7-9e86-62b09ab69bee" containerName="revoke-keys" Nov 03 22:28:43.585831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.585730 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c19a6f7-87bc-45e7-9e86-62b09ab69bee" containerName="revoke-keys" Nov 03 22:28:43.585831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.585749 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" containerName="test-curl-7257-044416" Nov 03 22:28:43.592013 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.591009 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.621260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.621210 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlcs4\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-kube-api-access-wlcs4\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.621603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.621578 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.622201 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.622181 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.622439 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.622414 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.622661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.622640 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.623022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.623000 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.623234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.623193 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.623385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.623265 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.623385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.623303 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.627833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.627805 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:43.627980 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.627887 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l"] Nov 03 22:28:43.629281 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.629133 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:43.631365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.631150 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:28:43.724608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.724568 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.724817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.724653 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlcs4\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-kube-api-access-wlcs4\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.724817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.724735 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.724944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.724830 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.724944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.724855 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.725056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.724951 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.725155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.725134 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.725232 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.725214 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.725288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.725266 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.725687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.725665 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.725922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.725905 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.726474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.726449 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.727540 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.727508 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.727936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.727910 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.729829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.729782 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.729963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.729944 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.752560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.747386 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlcs4\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-kube-api-access-wlcs4\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.752560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.747882 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:43.918357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:43.918324 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:44.150805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.150771 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-044416" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" containerName="test-curl-7257-044416" containerID="cri-o://aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4" gracePeriod=2 Nov 03 22:28:44.163473 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:44.163438 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-044416\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:44.181956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.181924 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5"] Nov 03 22:28:44.202147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.202121 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.214769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.214630 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5"] Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232104 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrzl\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-kube-api-access-zvrzl\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232271 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232311 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232349 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232405 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232433 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232529 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232634 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.232758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.232764 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeb8542-d030-42da-a1fe-878fbc13bb73-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.261548 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:44.261401 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-044416\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:44.261548 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:44.261490 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b65df3_8781_4fc4_be9d_8c1df26fa290.slice/crio-143199237e6f9407532339b3cd18e4eecdfdd14cfc153a644b544ab37c6d4455 WatchSource:0}: Error finding container 143199237e6f9407532339b3cd18e4eecdfdd14cfc153a644b544ab37c6d4455: Status 404 returned error can't find the container with id 143199237e6f9407532339b3cd18e4eecdfdd14cfc153a644b544ab37c6d4455 Nov 03 22:28:44.265495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.265057 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:44.265495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.265120 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:44.265495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.265151 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:44.270656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.268921 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l"] Nov 03 22:28:44.360802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.360684 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeb8542-d030-42da-a1fe-878fbc13bb73-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.360970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.360840 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrzl\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-kube-api-access-zvrzl\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.361058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.360962 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.361058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.361006 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.361058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.361036 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.361233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.361095 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.361233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.361125 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.361233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.361189 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.361392 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.361306 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.377748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.367731 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeb8542-d030-42da-a1fe-878fbc13bb73-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.377748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.369579 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.377748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.369896 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.391014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.382548 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.391014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.385133 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.391014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.387216 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.394693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.394175 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.397213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.397182 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrzl\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-kube-api-access-zvrzl\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.407166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.404569 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.440930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.440905 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:44.453639 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:44.453603 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-044416\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:44.462985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.462954 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28101e0f-0721-4f73-b446-6de97f0b7997-kube-api-access-nfd5p\" (UniqueName: \"kubernetes.io/projected/28101e0f-0721-4f73-b446-6de97f0b7997-kube-api-access-nfd5p\") pod \"28101e0f-0721-4f73-b446-6de97f0b7997\" (UID: \"28101e0f-0721-4f73-b446-6de97f0b7997\") " Nov 03 22:28:44.465791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.465761 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28101e0f-0721-4f73-b446-6de97f0b7997-kube-api-access-nfd5p" pod "28101e0f-0721-4f73-b446-6de97f0b7997" (UID: "28101e0f-0721-4f73-b446-6de97f0b7997"). InnerVolumeSpecName "kube-api-access-nfd5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:44.528590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.528557 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:44.565893 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.565612 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfd5p\" (UniqueName: \"kubernetes.io/projected/28101e0f-0721-4f73-b446-6de97f0b7997-kube-api-access-nfd5p\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:28:44.730699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.730618 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28101e0f-0721-4f73-b446-6de97f0b7997" path="/var/lib/kubelet/pods/28101e0f-0721-4f73-b446-6de97f0b7997/volumes" Nov 03 22:28:44.881000 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:44.880946 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aeb8542_d030_42da_a1fe_878fbc13bb73.slice/crio-057d4641c34228730f6fb4dfaead0844e176c9b5634ffd5a9dd3b577223b8182 WatchSource:0}: Error finding container 057d4641c34228730f6fb4dfaead0844e176c9b5634ffd5a9dd3b577223b8182: Status 404 returned error can't find the container with id 057d4641c34228730f6fb4dfaead0844e176c9b5634ffd5a9dd3b577223b8182 Nov 03 22:28:44.883022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.882986 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5"] Nov 03 22:28:44.885036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.884989 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:44.885164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.885078 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:44.885164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:44.885121 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:45.181149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.181112 4625 generic.go:356] "Generic (PLEG): container finished" podID="28101e0f-0721-4f73-b446-6de97f0b7997" containerID="a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1" exitCode=137 Nov 03 22:28:45.181371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.181207 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-049167" Nov 03 22:28:45.181574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.181214 4625 scope.go:122] "RemoveContainer" containerID="a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1" Nov 03 22:28:45.185153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.184155 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" event={"ID":"d5b65df3-8781-4fc4-be9d-8c1df26fa290","Type":"ContainerStarted","Data":"95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1"} Nov 03 22:28:45.185153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.184200 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" event={"ID":"d5b65df3-8781-4fc4-be9d-8c1df26fa290","Type":"ContainerStarted","Data":"143199237e6f9407532339b3cd18e4eecdfdd14cfc153a644b544ab37c6d4455"} Nov 03 22:28:45.187912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.187887 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" event={"ID":"3aeb8542-d030-42da-a1fe-878fbc13bb73","Type":"ContainerStarted","Data":"475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc"} Nov 03 22:28:45.188044 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.187932 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" event={"ID":"3aeb8542-d030-42da-a1fe-878fbc13bb73","Type":"ContainerStarted","Data":"057d4641c34228730f6fb4dfaead0844e176c9b5634ffd5a9dd3b577223b8182"} Nov 03 22:28:45.202987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.202962 4625 scope.go:122] "RemoveContainer" containerID="a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1" Nov 03 22:28:45.203351 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:45.203332 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1\": container with ID starting with a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1 not found: ID does not exist" containerID="a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1" Nov 03 22:28:45.203460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.203442 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1"} err="failed to get container status \"a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1\": rpc error: code = NotFound desc = could not find container \"a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1\": container with ID starting with a890c2f7311872e14d0c7fe96a9d2b864393f84662116e91bd1366d82c58a1f1 not found: ID does not exist" Nov 03 22:28:45.230582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.230522 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podStartSLOduration=2.230505015 podStartE2EDuration="2.230505015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:45.227108144 +0000 UTC m=+946.656444137" watchObservedRunningTime="2026-11-03 22:28:45.230505015 +0000 UTC m=+946.659841065" Nov 03 22:28:45.281754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.281656 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podStartSLOduration=1.281641917 podStartE2EDuration="1.281641917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:45.2770116 +0000 UTC m=+946.706347595" watchObservedRunningTime="2026-11-03 22:28:45.281641917 +0000 UTC m=+946.710977910" Nov 03 22:28:45.529104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.529069 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:45.531155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.531120 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:45.531373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.531348 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:45.918768 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.918737 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:45.920921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.920890 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:28:45.921054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:45.920942 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:28:46.495838 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:46.492744 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6af996a6_0fda_45ff_aed7_13a9873fdbe5.slice/crio-aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6af996a6_0fda_45ff_aed7_13a9873fdbe5.slice/crio-conmon-aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:28:46.532335 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:46.532295 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:46.532536 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:46.532359 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:46.900296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:46.900269 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:46.919277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:46.919248 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:28:46.919620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:46.919302 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:28:47.021507 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.021214 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af996a6-0fda-45ff-aed7-13a9873fdbe5-kube-api-access-cb2dl\" (UniqueName: \"kubernetes.io/projected/6af996a6-0fda-45ff-aed7-13a9873fdbe5-kube-api-access-cb2dl\") pod \"6af996a6-0fda-45ff-aed7-13a9873fdbe5\" (UID: \"6af996a6-0fda-45ff-aed7-13a9873fdbe5\") " Nov 03 22:28:47.023972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.023945 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af996a6-0fda-45ff-aed7-13a9873fdbe5-kube-api-access-cb2dl" pod "6af996a6-0fda-45ff-aed7-13a9873fdbe5" (UID: "6af996a6-0fda-45ff-aed7-13a9873fdbe5"). InnerVolumeSpecName "kube-api-access-cb2dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:47.122946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.122908 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2dl\" (UniqueName: \"kubernetes.io/projected/6af996a6-0fda-45ff-aed7-13a9873fdbe5-kube-api-access-cb2dl\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:28:47.288740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.287807 4625 generic.go:356] "Generic (PLEG): container finished" podID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" containerID="aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4" exitCode=137 Nov 03 22:28:47.288942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.288765 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:47.292734 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.290591 4625 scope.go:122] "RemoveContainer" containerID="aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4" Nov 03 22:28:47.321811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.321788 4625 scope.go:122] "RemoveContainer" containerID="aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4" Nov 03 22:28:47.322251 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:47.322198 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4\": container with ID starting with aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4 not found: ID does not exist" containerID="aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4" Nov 03 22:28:47.322251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.322226 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4"} err="failed to get container status \"aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4\": rpc error: code = NotFound desc = could not find container \"aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4\": container with ID starting with aacda8692c1030009352b95af0cf77ec3de96dc55ca33b3828bfe6cc5b52b8a4 not found: ID does not exist" Nov 03 22:28:47.331645 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:47.331613 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-044416\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:47.529158 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.529066 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:47.529158 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.529126 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:47.919367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.919331 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:28:47.919826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:47.919381 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:28:48.529783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:48.529744 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:48.529937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:48.529808 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:48.732480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:48.732442 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" path="/var/lib/kubelet/pods/6af996a6-0fda-45ff-aed7-13a9873fdbe5/volumes" Nov 03 22:28:48.747990 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:28:48.747945 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-044416\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="6af996a6-0fda-45ff-aed7-13a9873fdbe5" pod="opendatahub/test-curl-7257-044416" Nov 03 22:28:48.919026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:48.918975 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:28:48.919204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:48.919042 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:28:49.512835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.511057 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd"] Nov 03 22:28:49.517921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.517897 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.546126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.531272 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:49.546126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.531315 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:49.564297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.561246 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b7b10505-gw-openshift-default-dockercfg-58jlf\"" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.609476 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-socket\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.609558 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-envoy\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.609674 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-data\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.609766 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-podinfo\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.609848 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdk6n\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-kube-api-access-mdk6n\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.610018 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-certs\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.610135 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-token\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.610280 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istiod-ca-cert\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.615739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.610336 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-credential-socket\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.621658 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.620820 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd"] Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.778844 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istiod-ca-cert\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.778923 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-credential-socket\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.778965 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-socket\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.779037 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-envoy\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.779129 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-data\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.779243 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-podinfo\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.779308 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdk6n\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-kube-api-access-mdk6n\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.779437 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-certs\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.779515 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-token\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.780787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.780414 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-credential-socket\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.781402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.781215 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-socket\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.927263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.850105 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-data\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.927263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.850675 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-certs\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.927263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.926491 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istiod-ca-cert\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.941979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.941514 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:28:49.941979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.941570 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:28:49.959196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.959127 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdk6n\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-kube-api-access-mdk6n\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:49.996396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:49.996361 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-podinfo\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:50.089046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.086512 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-token\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:50.089883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.089854 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-envoy\") pod \"e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:50.197707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.197677 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:50.529690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.529643 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:50.530161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.529704 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:50.894194 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:50.894156 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee156b68_b18c_48a0_ba5e_5ddac259c3b9.slice/crio-1a8f0887c7f2fc68fb7254e8e56b69c1750e84ff193e0d0666d3308dc4c941b7 WatchSource:0}: Error finding container 1a8f0887c7f2fc68fb7254e8e56b69c1750e84ff193e0d0666d3308dc4c941b7: Status 404 returned error can't find the container with id 1a8f0887c7f2fc68fb7254e8e56b69c1750e84ff193e0d0666d3308dc4c941b7 Nov 03 22:28:50.897324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.897283 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:50.897432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.897398 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:50.897480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.897464 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:50.897655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.897637 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd"] Nov 03 22:28:50.919050 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.919027 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:28:50.919141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:50.919070 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:28:51.526993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:51.523609 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" event={"ID":"ee156b68-b18c-48a0-ba5e-5ddac259c3b9","Type":"ContainerStarted","Data":"18bdf1d1f73698d5395fba91b717f7c54d1336b39589fd62f61c5f090e63b5cb"} Nov 03 22:28:51.526993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:51.523677 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" event={"ID":"ee156b68-b18c-48a0-ba5e-5ddac259c3b9","Type":"ContainerStarted","Data":"1a8f0887c7f2fc68fb7254e8e56b69c1750e84ff193e0d0666d3308dc4c941b7"} Nov 03 22:28:51.529460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:51.529430 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:51.529606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:51.529488 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:51.604551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:51.604488 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" podStartSLOduration=2.604467638 podStartE2EDuration="2.604467638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:51.587657461 +0000 UTC m=+953.016993455" watchObservedRunningTime="2026-11-03 22:28:51.604467638 +0000 UTC m=+953.033803638" Nov 03 22:28:51.928231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:51.928199 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:28:51.928594 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:51.928552 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:28:52.198745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.198639 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:52.206046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.206016 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:52.531797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.529776 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:28:52.531797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.529847 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:28:52.537526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.537487 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:52.539333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.539083 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:28:52.925849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.925819 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:52.926326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.926166 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:52.926951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:52.926930 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:28:53.533870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.533842 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:53.534156 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.534137 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:53.535093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.535047 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:28:53.626290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.626261 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88"] Nov 03 22:28:53.641885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.641859 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.665948 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.665694 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1b279244-openshift-default-dockercfg-h625r\"" Nov 03 22:28:53.669389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.669359 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88"] Nov 03 22:28:53.725183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725148 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-podinfo\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725196 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsp8\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-kube-api-access-9jsp8\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725221 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-socket\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725373 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b963989a-255c-41b8-84a3-f3ffc60141ac-istiod-ca-cert\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725420 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-credential-socket\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725452 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-certs\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725526 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-token\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725609 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725593 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-envoy\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.725643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.725633 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-data\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.826756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.826630 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-data\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.826953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.826809 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-podinfo\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.826953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.826872 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsp8\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-kube-api-access-9jsp8\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.826953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.826905 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-socket\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.827129 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.827067 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b963989a-255c-41b8-84a3-f3ffc60141ac-istiod-ca-cert\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.827188 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.827150 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-data\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.827813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.827783 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-socket\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.828124 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.828103 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-credential-socket\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.828354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.827147 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-credential-socket\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.828826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.828803 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b963989a-255c-41b8-84a3-f3ffc60141ac-istiod-ca-cert\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.829057 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.829032 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-certs\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.829138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.829106 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-token\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.829196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.829161 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-envoy\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.830420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.830394 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-certs\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.834991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.831856 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-podinfo\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.877947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.871769 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-envoy\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.885384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.885314 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-token\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.928507 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.913528 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsp8\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-kube-api-access-9jsp8\") pod \"e2e-ait-1b279244-openshift-default-74dd88dd99-6px88\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:53.966270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:53.965994 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:54.517155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:54.517123 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88"] Nov 03 22:28:54.517894 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:54.517871 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb963989a_255c_41b8_84a3_f3ffc60141ac.slice/crio-c9cf66d35512f0458d78c720bce07478c0075caa279745771d859f1f855eba8c WatchSource:0}: Error finding container c9cf66d35512f0458d78c720bce07478c0075caa279745771d859f1f855eba8c: Status 404 returned error can't find the container with id c9cf66d35512f0458d78c720bce07478c0075caa279745771d859f1f855eba8c Nov 03 22:28:54.521199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:54.521155 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:54.521316 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:54.521244 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:54.521316 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:54.521287 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:54.554777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:54.554747 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" event={"ID":"b963989a-255c-41b8-84a3-f3ffc60141ac","Type":"ContainerStarted","Data":"c9cf66d35512f0458d78c720bce07478c0075caa279745771d859f1f855eba8c"} Nov 03 22:28:55.563811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.563776 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" event={"ID":"b963989a-255c-41b8-84a3-f3ffc60141ac","Type":"ContainerStarted","Data":"163585c2a9aa501781237492aa63e5c3d02118c4517d9dacd820a62d97768de9"} Nov 03 22:28:55.629747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.628298 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" podStartSLOduration=2.628276143 podStartE2EDuration="2.628276143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:55.626349735 +0000 UTC m=+957.055685751" watchObservedRunningTime="2026-11-03 22:28:55.628276143 +0000 UTC m=+957.057612137" Nov 03 22:28:55.881546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.881510 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh"] Nov 03 22:28:55.886514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.886483 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:55.902858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.902830 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b7b10505\"" Nov 03 22:28:55.905763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.905729 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh"] Nov 03 22:28:55.957157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.957118 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m842\" (UniqueName: \"kubernetes.io/projected/32665d6c-ab04-4675-81eb-4388bc4e6acc-kube-api-access-8m842\") pod \"payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:55.957433 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.957398 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32665d6c-ab04-4675-81eb-4388bc4e6acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:55.967541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.967514 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:55.974369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:55.974343 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:56.105739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.094175 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4"] Nov 03 22:28:56.105739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.098470 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32665d6c-ab04-4675-81eb-4388bc4e6acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:56.105739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.098667 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m842\" (UniqueName: \"kubernetes.io/projected/32665d6c-ab04-4675-81eb-4388bc4e6acc-kube-api-access-8m842\") pod \"payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:56.105739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.099751 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.106074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.106022 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32665d6c-ab04-4675-81eb-4388bc4e6acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:56.129469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.128847 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b7b10505-dockercfg-lwcvh\"" Nov 03 22:28:56.223737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.207615 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419ba9-cecb-4d9c-9466-2d8d42245508-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.261273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.255874 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97b4g\" (UniqueName: \"kubernetes.io/projected/bd419ba9-cecb-4d9c-9466-2d8d42245508-kube-api-access-97b4g\") pod \"payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.262210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.262143 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m842\" (UniqueName: \"kubernetes.io/projected/32665d6c-ab04-4675-81eb-4388bc4e6acc-kube-api-access-8m842\") pod \"payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:56.367826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.367790 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4"] Nov 03 22:28:56.370976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.369254 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419ba9-cecb-4d9c-9466-2d8d42245508-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.371332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.371306 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97b4g\" (UniqueName: \"kubernetes.io/projected/bd419ba9-cecb-4d9c-9466-2d8d42245508-kube-api-access-97b4g\") pod \"payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.372918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.372893 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419ba9-cecb-4d9c-9466-2d8d42245508-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.420806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.420333 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97b4g\" (UniqueName: \"kubernetes.io/projected/bd419ba9-cecb-4d9c-9466-2d8d42245508-kube-api-access-97b4g\") pod \"payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.506711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.506676 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:56.582997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.580278 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:56.596307 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.596184 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:56.597446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.597428 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:28:56.834541 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:56.834511 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32665d6c_ab04_4675_81eb_4388bc4e6acc.slice/crio-fba4d14804505cc34cd8da633e44acb4f801f3f91d229524fe5cbaab43168f5a WatchSource:0}: Error finding container fba4d14804505cc34cd8da633e44acb4f801f3f91d229524fe5cbaab43168f5a: Status 404 returned error can't find the container with id fba4d14804505cc34cd8da633e44acb4f801f3f91d229524fe5cbaab43168f5a Nov 03 22:28:56.838290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.837933 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh"] Nov 03 22:28:56.914110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:56.914081 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4"] Nov 03 22:28:56.915177 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:28:56.915150 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd419ba9_cecb_4d9c_9466_2d8d42245508.slice/crio-5830ee6e3e1282fcf04936ab17f7d400c4277e5ad13dc10128d0f7f965015824 WatchSource:0}: Error finding container 5830ee6e3e1282fcf04936ab17f7d400c4277e5ad13dc10128d0f7f965015824: Status 404 returned error can't find the container with id 5830ee6e3e1282fcf04936ab17f7d400c4277e5ad13dc10128d0f7f965015824 Nov 03 22:28:57.606357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:57.605633 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" event={"ID":"32665d6c-ab04-4675-81eb-4388bc4e6acc","Type":"ContainerStarted","Data":"792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11"} Nov 03 22:28:57.606357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:57.605706 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" event={"ID":"32665d6c-ab04-4675-81eb-4388bc4e6acc","Type":"ContainerStarted","Data":"fba4d14804505cc34cd8da633e44acb4f801f3f91d229524fe5cbaab43168f5a"} Nov 03 22:28:57.606357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:57.605754 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:28:57.607839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:57.607810 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" event={"ID":"bd419ba9-cecb-4d9c-9466-2d8d42245508","Type":"ContainerStarted","Data":"43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f"} Nov 03 22:28:57.608026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:57.607999 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" event={"ID":"bd419ba9-cecb-4d9c-9466-2d8d42245508","Type":"ContainerStarted","Data":"5830ee6e3e1282fcf04936ab17f7d400c4277e5ad13dc10128d0f7f965015824"} Nov 03 22:28:57.646029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:57.645969 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" podStartSLOduration=2.645950454 podStartE2EDuration="2.645950454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:57.644996479 +0000 UTC m=+959.074332487" watchObservedRunningTime="2026-11-03 22:28:57.645950454 +0000 UTC m=+959.075286450" Nov 03 22:28:57.711472 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:57.711416 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" podStartSLOduration=2.71140173 podStartE2EDuration="2.71140173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:57.708654687 +0000 UTC m=+959.137990681" watchObservedRunningTime="2026-11-03 22:28:57.71140173 +0000 UTC m=+959.140737723" Nov 03 22:28:58.419179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.419146 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75"] Nov 03 22:28:58.423865 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.423836 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.463339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.463305 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75"] Nov 03 22:28:58.507238 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507208 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507238 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507242 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507339 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507398 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507536 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507463 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc89k\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-kube-api-access-nc89k\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507536 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507489 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507596 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507654 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.507727 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.507702 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609241 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609203 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609241 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609251 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609336 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609394 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609463 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc89k\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-kube-api-access-nc89k\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609507 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609647 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609742 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609773 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609752 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.609882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.609835 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.610435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.610191 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.610435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.610211 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.610435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.610196 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.612231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.612205 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.613054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.613026 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.616486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.616460 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:28:58.636903 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.636867 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc89k\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-kube-api-access-nc89k\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.637057 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.636966 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:58.755612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:58.755533 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:28:59.095315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.095283 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75"] Nov 03 22:28:59.102566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.102177 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:59.102566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.102318 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:59.102566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.102412 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:28:59.306821 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.304624 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2"] Nov 03 22:28:59.314811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.313915 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.343819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.340391 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m"] Nov 03 22:28:59.348013 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.347950 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.377680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.376829 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2"] Nov 03 22:28:59.383930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.383903 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m"] Nov 03 22:28:59.427428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427304 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.427428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427343 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427431 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427458 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427574 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427597 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427627 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427654 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427673 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2c26\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-kube-api-access-z2c26\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.427703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427692 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427756 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427787 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427818 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427856 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427874 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b397c3-cce1-40d0-bd67-46c2b709bb99-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427895 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427923 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfjj\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-kube-api-access-vdfjj\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.428172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.427976 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.528937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.528898 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.528945 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.528982 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529008 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529035 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2c26\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-kube-api-access-z2c26\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529071 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529130 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529175 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529229 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529292 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529324 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b397c3-cce1-40d0-bd67-46c2b709bb99-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529358 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529388 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529408 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfjj\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-kube-api-access-vdfjj\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529495 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529531 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529568 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529593 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529610 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529626 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.529973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529636 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.529973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529854 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.530340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.529983 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.530397 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.530348 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.530455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.530403 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.530593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.530571 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.530705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.530649 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.530910 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.530884 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b397c3-cce1-40d0-bd67-46c2b709bb99-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.532165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.532138 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.534741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.534700 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.534833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.534766 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.534998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.534980 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.553843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.553816 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.554034 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.554009 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2c26\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-kube-api-access-z2c26\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.554079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.554047 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfjj\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-kube-api-access-vdfjj\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.555261 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.555235 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.632502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.632390 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" event={"ID":"00f8e9cc-c495-4291-86c5-d64db5dd7af9","Type":"ContainerStarted","Data":"b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a"} Nov 03 22:28:59.633155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.633089 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" event={"ID":"00f8e9cc-c495-4291-86c5-d64db5dd7af9","Type":"ContainerStarted","Data":"ef5472693d219363e64221aa89e5cba67a10b1e12e6151cbbc197e987bb5ff4a"} Nov 03 22:28:59.678183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.678147 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:28:59.680968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.680915 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podStartSLOduration=1.6809010089999998 podStartE2EDuration="1.680901009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:59.676028753 +0000 UTC m=+961.105364752" watchObservedRunningTime="2026-11-03 22:28:59.680901009 +0000 UTC m=+961.110237002" Nov 03 22:28:59.695355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.695325 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:28:59.755910 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:28:59.755883 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:29:00.078884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.078855 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2"] Nov 03 22:29:00.080871 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.080837 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:00.080998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.080925 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:00.080998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.080975 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:00.083141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.083089 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m"] Nov 03 22:29:00.085996 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.085964 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:00.086094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.086040 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:00.086094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.086084 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:00.682741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.682442 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" event={"ID":"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a","Type":"ContainerStarted","Data":"cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4"} Nov 03 22:29:00.682741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.682533 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" event={"ID":"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a","Type":"ContainerStarted","Data":"781b4f4a4a6c3129b934bbaa48e2a2e2c5e8f735342cff13b4daf36af13f06c2"} Nov 03 22:29:00.686896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.686866 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" event={"ID":"86b397c3-cce1-40d0-bd67-46c2b709bb99","Type":"ContainerStarted","Data":"8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd"} Nov 03 22:29:00.687088 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.687071 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" event={"ID":"86b397c3-cce1-40d0-bd67-46c2b709bb99","Type":"ContainerStarted","Data":"b4fd955e72e4152caf8d18980497dcdcd6c196bdfa327182367d2fa6adb0c7d5"} Nov 03 22:29:00.695762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.695708 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:29:00.726376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.726316 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podStartSLOduration=1.726298061 podStartE2EDuration="1.726298061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:00.725289488 +0000 UTC m=+962.154625482" watchObservedRunningTime="2026-11-03 22:29:00.726298061 +0000 UTC m=+962.155634056" Nov 03 22:29:00.771971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.760507 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:00.771971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.760567 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:00.786676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:00.784929 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podStartSLOduration=1.784905981 podStartE2EDuration="1.784905981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:00.773276841 +0000 UTC m=+962.202612835" watchObservedRunningTime="2026-11-03 22:29:00.784905981 +0000 UTC m=+962.214241975" Nov 03 22:29:02.368407 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.221932 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:29:02.389502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.300940 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.389829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.389797 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:02.433006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.307996 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.433006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.431395 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:02.444363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.335834 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.444650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.444620 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:02.454643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.454611 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4"] Nov 03 22:29:02.650837 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.635774 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:02.669932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.653256 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4"] Nov 03 22:29:02.703862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.664338 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b279244-dockercfg-d2wmh\"" Nov 03 22:29:02.703862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.698095 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b279244\"" Nov 03 22:29:02.704173 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.703997 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.704173 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.704047 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:02.729741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.704670 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.730082 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.730029 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:02.756740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.756392 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.756740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.756461 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:02.822794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.808622 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f0bb48-07ef-4193-80ae-004397ef48b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:02.822794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.808757 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2ql\" (UniqueName: \"kubernetes.io/projected/84f0bb48-07ef-4193-80ae-004397ef48b7-kube-api-access-vl2ql\") pod \"payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:02.892742 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:02.892005 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:29:02.918526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.918419 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2ql\" (UniqueName: \"kubernetes.io/projected/84f0bb48-07ef-4193-80ae-004397ef48b7-kube-api-access-vl2ql\") pod \"payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:02.919088 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.919057 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f0bb48-07ef-4193-80ae-004397ef48b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:02.927748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.927233 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f0bb48-07ef-4193-80ae-004397ef48b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:02.986750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:02.984901 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2ql\" (UniqueName: \"kubernetes.io/projected/84f0bb48-07ef-4193-80ae-004397ef48b7-kube-api-access-vl2ql\") pod \"payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:03.061017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:03.060978 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:03.725741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:03.707242 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:03.725741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:03.707304 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:03.744781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:03.737818 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:03.744781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:03.737878 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:03.760471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:03.759510 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:03.760471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:03.759565 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:04.323741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.315153 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4"] Nov 03 22:29:04.376766 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:29:04.333209 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f0bb48_07ef_4193_80ae_004397ef48b7.slice/crio-d9acd333311f724c7b969939dbbaabdc01298f5f2c54ad698c63d958a55a4041 WatchSource:0}: Error finding container d9acd333311f724c7b969939dbbaabdc01298f5f2c54ad698c63d958a55a4041: Status 404 returned error can't find the container with id d9acd333311f724c7b969939dbbaabdc01298f5f2c54ad698c63d958a55a4041 Nov 03 22:29:04.716737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.715190 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:04.716737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.715262 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:04.723747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.723112 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:04.723747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.723166 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:04.780424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.779609 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:04.780424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.779670 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:04.835742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:04.815565 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" event={"ID":"84f0bb48-07ef-4193-80ae-004397ef48b7","Type":"ContainerStarted","Data":"d9acd333311f724c7b969939dbbaabdc01298f5f2c54ad698c63d958a55a4041"} Nov 03 22:29:05.523950 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.523819 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:05.679160 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.679116 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:05.679423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.679389 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:05.702312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.702277 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:05.703959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.703904 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:05.757737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.757092 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:05.757737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.757149 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:05.928332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.928274 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" event={"ID":"84f0bb48-07ef-4193-80ae-004397ef48b7","Type":"ContainerStarted","Data":"74b71df6fc80cb9cc6d54c4dd46a06d2a599a0e2d8e2a9187dd218c392376949"} Nov 03 22:29:05.930585 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:05.930555 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:06.695750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:06.695639 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:06.695750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:06.695702 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:06.711735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:06.696573 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:06.712005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:06.711967 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:06.832208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:06.825291 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:06.832208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:06.825367 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:07.704971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:07.704926 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:07.720756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:07.705528 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:07.733777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:07.728296 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:07.733777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:07.728357 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:07.912753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:07.912682 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:07.912951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:07.912770 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:08.195638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.195593 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:08.694278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.694219 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:08.697558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.696825 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:08.697558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.696884 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:08.699164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.699131 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:08.712328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.712290 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:29:08.772759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.736994 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:08.798328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.798269 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" podStartSLOduration=7.798248922 podStartE2EDuration="7.798248922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:06.000787466 +0000 UTC m=+967.430123487" watchObservedRunningTime="2026-11-03 22:29:08.798248922 +0000 UTC m=+970.227584917" Nov 03 22:29:08.861832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:08.861790 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:09.025703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.023957 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.025703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.024017 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:29:09.206894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.206837 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:09.220742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.208155 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:29:09.221551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.221525 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9"] Nov 03 22:29:09.592776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.589291 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:09.616769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.614318 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:09.767030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.746187 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:09.770739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.767896 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:29:09.789883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.789845 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-51bfa2e1-openshift-default-dockercfg-gcl7f\"" Nov 03 22:29:09.827391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.803662 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.827391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.803751 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:09.827391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.823462 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.827391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.823535 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:09.871131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.871018 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:09.872260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.872239 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:29:09.916029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.915982 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-certs\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:09.916451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.916426 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-socket\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:09.916771 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.916750 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60521624-9dd2-42e5-b0d0-9fd65d597179-istiod-ca-cert\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:09.917179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.917158 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-token\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:09.917357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.917334 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-podinfo\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:09.927913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.917481 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-data\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:09.948740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.929379 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-credential-socket\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.052119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.953862 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzmj\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-kube-api-access-wjzmj\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.052119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:09.953937 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-envoy\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.052119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.028892 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:29:10.217120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.216847 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.250809 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-token\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.250883 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-podinfo\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.250932 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-data\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.250976 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-credential-socket\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.251029 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzmj\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-kube-api-access-wjzmj\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.251075 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-envoy\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.251198 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-certs\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.251337 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-socket\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.277762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.251495 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60521624-9dd2-42e5-b0d0-9fd65d597179-istiod-ca-cert\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.570801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.530339 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-data\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.819978 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.556437 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-credential-socket\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:10.820592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.557613 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9"] Nov 03 22:29:11.054744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.050765 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-envoy\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.116743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.623497 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.260391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.675030 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-certs\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.321576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.675593 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-podinfo\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.321877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.675843 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-socket\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.322048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.851899 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:11.322224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.322182 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.322943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.852566 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-token\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.323080 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.598397 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60521624-9dd2-42e5-b0d0-9fd65d597179-istiod-ca-cert\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.323204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.260068 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzmj\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-kube-api-access-wjzmj\") pod \"e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.323305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.260789 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.323763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.852015 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.323914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.323890 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:11.324030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.060580 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.324146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.324127 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:11.324289 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:10.883264 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:11.348009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.347966 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.481079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.449932 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:11.929744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:11.925186 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:29:12.070584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.070489 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.070862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.070828 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:12.082754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.082686 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.083018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.082991 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:12.269940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.269860 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.368801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.318785 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.368801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.327806 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.368801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.328941 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:12.411217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.411179 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:12.411460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.411431 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.416739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.413410 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.706156 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.706076 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.706375 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.706346 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:12.757908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.757870 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.758172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:12.758146 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:13.297774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.297730 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.343388 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.343282 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.353181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.345146 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.421163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.421115 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.682478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.682446 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.694441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.682742 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:13.695153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.694669 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.695770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.695748 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.695922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.695903 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:13.696793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.694405 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:29:13.788751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:13.761535 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:29:14.349801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.309800 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:14.376026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.374799 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:14.376026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.375221 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.494749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.446708 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:14.798879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.693048 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:29:14.798879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.779730 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.864198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.752049 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:14.864482 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.864453 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.866464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.781036 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.866657 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.866635 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:14.885614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.885576 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:14.885912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:14.885880 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.026876 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.019292 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Nov 03 22:29:15.367834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.356414 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:15.367834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.356842 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:15.367834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.357174 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:15.452743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.432876 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.452743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.434446 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:15.486691 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.486647 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:15.792743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.771888 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" podUID="2d8e5e86-4926-41f1-ba99-b2da2fab4889" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.926740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.796019 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.980742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.975764 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.980742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:15.975823 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:29:16.174796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.084218 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:29:16.174796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.108865 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:29:16.294342 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:16.265307 4625 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.491s" Nov 03 22:29:16.294342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.265476 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:29:16.294342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.265520 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9"] Nov 03 22:29:16.437376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.437174 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.438641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.438607 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:16.729794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.712843 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:16.874538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.874499 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:17.028528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.028425 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:17.047121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:16.942802 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:17.305423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.305298 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:17.305693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.305663 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:17.453752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.446824 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:17.453752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.447238 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:17.727739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.716257 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:17.826205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.821050 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:17.826205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.821119 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:17.979342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:17.969508 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm"] Nov 03 22:29:18.066351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.050546 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:18.066351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.051150 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.067333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.067301 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1b279244-serving-cert\"" Nov 03 22:29:18.107739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.087255 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:18.107739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.087317 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.107739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.088600 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" event={"ID":"60521624-9dd2-42e5-b0d0-9fd65d597179","Type":"ContainerStarted","Data":"6022561e80937acdaadc1b11b056d481132f4eee5d61033987915ccba1a6238c"} Nov 03 22:29:18.171612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.171572 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a8793d-65b1-4529-acfd-c3ae76f32ba0-openshift-service-ca\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.171908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.171884 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wk56\" (UniqueName: \"kubernetes.io/projected/03a8793d-65b1-4529-acfd-c3ae76f32ba0-kube-api-access-8wk56\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.172126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.172107 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-metrics-certs\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.172407 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.172386 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.228497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.225352 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:29:18.240200 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.240107 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:29:18.262814 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.262776 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm"] Nov 03 22:29:18.278779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.274857 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.278779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.275066 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a8793d-65b1-4529-acfd-c3ae76f32ba0-openshift-service-ca\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.278779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.275112 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wk56\" (UniqueName: \"kubernetes.io/projected/03a8793d-65b1-4529-acfd-c3ae76f32ba0-kube-api-access-8wk56\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.278779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.275246 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-metrics-certs\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.280035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.280007 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a8793d-65b1-4529-acfd-c3ae76f32ba0-openshift-service-ca\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.305596 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:18.280868 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-serving-cert: secret "maas-api-e2e-ait-1b279244-serving-cert" not found Nov 03 22:29:18.305596 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:18.298208 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls podName:03a8793d-65b1-4529-acfd-c3ae76f32ba0 nodeName:}" failed. No retries permitted until 2026-11-03 22:29:18.798181786 +0000 UTC m=+980.227517771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls") pod "maas-api-e2e-ait-1b279244-589ccd554-dvjtm" (UID: "03a8793d-65b1-4529-acfd-c3ae76f32ba0") : secret "maas-api-e2e-ait-1b279244-serving-cert" not found Nov 03 22:29:18.378743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.348018 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-metrics-certs\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.378743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.359848 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:18.378743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.359918 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.472144 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.459245 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:18.472144 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.459640 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.642390 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.642358 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:18.642667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.642626 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.656880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.656827 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.779744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.775972 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wk56\" (UniqueName: \"kubernetes.io/projected/03a8793d-65b1-4529-acfd-c3ae76f32ba0-kube-api-access-8wk56\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.847745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.810562 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.929793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.914199 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:18.929793 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:18.914783 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-serving-cert: secret "maas-api-e2e-ait-1b279244-serving-cert" not found Nov 03 22:29:18.929793 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:18.914862 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls podName:03a8793d-65b1-4529-acfd-c3ae76f32ba0 nodeName:}" failed. No retries permitted until 2026-11-03 22:29:19.914833688 +0000 UTC m=+981.344169680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls") pod "maas-api-e2e-ait-1b279244-589ccd554-dvjtm" (UID: "03a8793d-65b1-4529-acfd-c3ae76f32ba0") : secret "maas-api-e2e-ait-1b279244-serving-cert" not found Nov 03 22:29:18.997739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.970565 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:18.997739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:18.970617 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:19.063643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.063599 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:19.446740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.431263 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:29:19.467970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.467917 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:19.468649 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.468600 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:19.506287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.494897 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:19.506287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.494957 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:19.792901 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.792806 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" event={"ID":"60521624-9dd2-42e5-b0d0-9fd65d597179","Type":"ContainerStarted","Data":"630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868"} Nov 03 22:29:19.793172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.793152 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:19.834096 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.834060 4625 patch_prober.go:29] interesting pod/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:29:19.834353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.834326 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" podUID="60521624-9dd2-42e5-b0d0-9fd65d597179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 22:29:19.933257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.933224 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:29:19.959921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:19.959890 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:20.012480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.012426 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:20.014177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.013655 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:20.033143 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.033103 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:20.104045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.103794 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:20.332503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.332468 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:29:20.362795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.341594 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\") pod \"maas-api-e2e-ait-1b279244-589ccd554-dvjtm\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:20.364690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.364624 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" podStartSLOduration=12.364600823 podStartE2EDuration="12.364600823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:19.894662558 +0000 UTC m=+981.323998553" watchObservedRunningTime="2026-11-03 22:29:20.364600823 +0000 UTC m=+981.793936817" Nov 03 22:29:20.418391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.387981 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:20.418391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.399117 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:29:20.468413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.468375 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:20.598888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.598837 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:20.628900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.628794 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:20.664742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.615213 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:29:20.769793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:20.756873 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:29:21.170369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:21.170337 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:29:21.239927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:21.239895 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:29:21.253760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:21.252568 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:21.253760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:21.252694 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:29:21.274817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:21.274785 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:21.304904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:21.275961 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:29:21.931376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:21.898501 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm"] Nov 03 22:29:22.684535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:22.684498 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" event={"ID":"03a8793d-65b1-4529-acfd-c3ae76f32ba0","Type":"ContainerStarted","Data":"34df7dd14b39d25a89be58b1173e364add9c0e172e01f4d2a127b4dc8c23c40b"} Nov 03 22:29:23.693759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:23.693684 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" event={"ID":"03a8793d-65b1-4529-acfd-c3ae76f32ba0","Type":"ContainerStarted","Data":"57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba"} Nov 03 22:29:23.694238 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:23.693798 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:23.756478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:23.756427 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" podStartSLOduration=5.953497083 podStartE2EDuration="6.756413351s" totalImagesPullingTime="802.916268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:23.751449368 +0000 UTC m=+985.180785361" watchObservedRunningTime="2026-11-03 22:29:23.756413351 +0000 UTC m=+985.185749344" Nov 03 22:29:24.812590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:24.812552 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j"] Nov 03 22:29:24.817615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:24.817595 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:24.834984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:24.834953 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j"] Nov 03 22:29:24.840293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:24.840267 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-51bfa2e1-dockercfg-2qct7\"" Nov 03 22:29:24.861595 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:24.861565 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-51bfa2e1\"" Nov 03 22:29:24.987428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:24.967961 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7b5g\" (UniqueName: \"kubernetes.io/projected/7fcdf9ce-4504-4143-8dea-84a702e155e9-kube-api-access-g7b5g\") pod \"payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:24.987952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:24.987918 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdf9ce-4504-4143-8dea-84a702e155e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:25.098813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:25.091903 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdf9ce-4504-4143-8dea-84a702e155e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:25.098813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:25.092218 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7b5g\" (UniqueName: \"kubernetes.io/projected/7fcdf9ce-4504-4143-8dea-84a702e155e9-kube-api-access-g7b5g\") pod \"payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:25.099638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:25.099023 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdf9ce-4504-4143-8dea-84a702e155e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:25.158613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:25.158570 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7b5g\" (UniqueName: \"kubernetes.io/projected/7fcdf9ce-4504-4143-8dea-84a702e155e9-kube-api-access-g7b5g\") pod \"payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:25.434036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:25.433939 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:25.851825 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:25.851701 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j"] Nov 03 22:29:26.729833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:26.729793 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" event={"ID":"7fcdf9ce-4504-4143-8dea-84a702e155e9","Type":"ContainerStarted","Data":"f938bfd5b0e8fb2154b3cfa097ae785167984be3b267c54f3e1888c8ec35d604"} Nov 03 22:29:26.730022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:26.729858 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:26.730022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:26.729876 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" event={"ID":"7fcdf9ce-4504-4143-8dea-84a702e155e9","Type":"ContainerStarted","Data":"7db8c0a81e0f1a132b6ff6fb17ff3ea3d71526f72ba1424b2b8d801278b2ae46"} Nov 03 22:29:26.770529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:26.770476 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" podStartSLOduration=2.7704620330000003 podStartE2EDuration="2.770462033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:26.768063529 +0000 UTC m=+988.197399528" watchObservedRunningTime="2026-11-03 22:29:26.770462033 +0000 UTC m=+988.199798026" Nov 03 22:29:28.767645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.767614 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn"] Nov 03 22:29:28.773208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.773183 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.809560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.809524 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn"] Nov 03 22:29:28.857241 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857200 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857262 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857282 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857364 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7sqq\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-kube-api-access-t7sqq\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857506 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857540 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857671 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857701 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.857880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.857804 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.995259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.994575 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.995259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.994663 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.995259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.994691 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.996558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.996448 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7sqq\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-kube-api-access-t7sqq\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.998878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.997236 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.999181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.999127 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:28.999377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.999315 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.001522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:28.999645 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.001522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.000554 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.001522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.001055 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.004269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.003661 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.004269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.003857 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.004269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.003963 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.005107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.005084 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.007629 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.007605 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9"] Nov 03 22:29:29.053791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.047224 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7sqq\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-kube-api-access-t7sqq\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.071252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.061114 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.095074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.089446 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.100377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.100328 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9"] Nov 03 22:29:29.101076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.101051 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.103365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.103340 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.111596 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.111574 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98"] Nov 03 22:29:29.123203 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.123180 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.126705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.126683 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98"] Nov 03 22:29:29.207126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207084 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrf4m\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-kube-api-access-lrf4m\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.207334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207143 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfgv8\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-kube-api-access-sfgv8\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.207334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207247 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.207464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207352 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.207464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207389 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.207464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207419 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.207584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207522 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c36897-4a20-4a43-a9b3-6874fe5af56e-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.207584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207566 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.207690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207662 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.207863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207696 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afc35dd-60f0-43d0-a3b4-055daea16d82-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.207863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207798 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.207982 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207883 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.208020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.207981 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.208020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.208016 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.208106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.208033 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.208106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.208060 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.208201 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.208116 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.208263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.208250 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.309349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309257 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.309349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309295 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.309349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309327 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.309641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309355 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.309641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309408 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.309641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309464 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrf4m\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-kube-api-access-lrf4m\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.309641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309492 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfgv8\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-kube-api-access-sfgv8\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.309641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309520 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.309641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309592 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.309641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309624 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309651 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309705 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c36897-4a20-4a43-a9b3-6874fe5af56e-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309801 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309839 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309878 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309915 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afc35dd-60f0-43d0-a3b4-055daea16d82-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309966 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310058 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310131 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310133 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.309964 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310219 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310696 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310477 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.310696 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310476 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310768 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310910 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.310988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.310964 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afc35dd-60f0-43d0-a3b4-055daea16d82-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.311427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.311400 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c36897-4a20-4a43-a9b3-6874fe5af56e-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.312028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.312007 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.312364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.312340 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.312558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.312540 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.312791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.312773 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.332063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.332024 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.332190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.332164 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfgv8\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-kube-api-access-sfgv8\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.332279 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.332256 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrf4m\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-kube-api-access-lrf4m\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.332814 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.332794 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.399442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.399402 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:29.443925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.443876 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:29.458092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.458064 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:29.708520 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.708490 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:29:29.743074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.743044 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn"] Nov 03 22:29:29.744246 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:29:29.744221 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d011ffd_3ab5_4b4a_b749_96a6a9d1eb20.slice/crio-4389d642ac9f50d8838bac656d0e6a9e6b4e2942e1eb83bdbe7a72cb783df512 WatchSource:0}: Error finding container 4389d642ac9f50d8838bac656d0e6a9e6b4e2942e1eb83bdbe7a72cb783df512: Status 404 returned error can't find the container with id 4389d642ac9f50d8838bac656d0e6a9e6b4e2942e1eb83bdbe7a72cb783df512 Nov 03 22:29:29.747518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.747487 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.747612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.747551 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.747612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.747578 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.758230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.758200 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" event={"ID":"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20","Type":"ContainerStarted","Data":"4389d642ac9f50d8838bac656d0e6a9e6b4e2942e1eb83bdbe7a72cb783df512"} Nov 03 22:29:29.828256 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.828202 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9"] Nov 03 22:29:29.829281 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:29:29.829249 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afc35dd_60f0_43d0_a3b4_055daea16d82.slice/crio-aa543dc3707ffe82fb35ac65813c2f4ea5fb915cbc0472f87037dda35fbb4afd WatchSource:0}: Error finding container aa543dc3707ffe82fb35ac65813c2f4ea5fb915cbc0472f87037dda35fbb4afd: Status 404 returned error can't find the container with id aa543dc3707ffe82fb35ac65813c2f4ea5fb915cbc0472f87037dda35fbb4afd Nov 03 22:29:29.833881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.833173 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.833881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.833252 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.833881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.833293 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.840977 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.840082 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98"] Nov 03 22:29:29.844778 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:29:29.844000 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c36897_4a20_4a43_a9b3_6874fe5af56e.slice/crio-0e6d4d66dfd39be1ce11e53f3a34c2427b50e64bc8ce2cc972a02805a41f44b7 WatchSource:0}: Error finding container 0e6d4d66dfd39be1ce11e53f3a34c2427b50e64bc8ce2cc972a02805a41f44b7: Status 404 returned error can't find the container with id 0e6d4d66dfd39be1ce11e53f3a34c2427b50e64bc8ce2cc972a02805a41f44b7 Nov 03 22:29:29.855266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.854941 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.855266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.855022 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:29.855266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:29.855064 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:29:31.012304 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.012265 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" event={"ID":"16c36897-4a20-4a43-a9b3-6874fe5af56e","Type":"ContainerStarted","Data":"6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52"} Nov 03 22:29:31.012908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.012882 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" event={"ID":"16c36897-4a20-4a43-a9b3-6874fe5af56e","Type":"ContainerStarted","Data":"0e6d4d66dfd39be1ce11e53f3a34c2427b50e64bc8ce2cc972a02805a41f44b7"} Nov 03 22:29:31.013066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.013042 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" event={"ID":"6afc35dd-60f0-43d0-a3b4-055daea16d82","Type":"ContainerStarted","Data":"916eaed96bb6e0b1924de82cfe05748c725615dabd868bbd36635da75cff65db"} Nov 03 22:29:31.013190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.013173 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" event={"ID":"6afc35dd-60f0-43d0-a3b4-055daea16d82","Type":"ContainerStarted","Data":"aa543dc3707ffe82fb35ac65813c2f4ea5fb915cbc0472f87037dda35fbb4afd"} Nov 03 22:29:31.013311 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.013298 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" event={"ID":"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20","Type":"ContainerStarted","Data":"085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247"} Nov 03 22:29:31.121186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.121117 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podStartSLOduration=3.121095262 podStartE2EDuration="3.121095262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:31.104059323 +0000 UTC m=+992.533395329" watchObservedRunningTime="2026-11-03 22:29:31.121095262 +0000 UTC m=+992.550431256" Nov 03 22:29:31.121885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.121837 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podStartSLOduration=3.121823724 podStartE2EDuration="3.121823724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:31.058146445 +0000 UTC m=+992.487482439" watchObservedRunningTime="2026-11-03 22:29:31.121823724 +0000 UTC m=+992.551159728" Nov 03 22:29:31.444746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.425792 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:31.447806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.447779 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:31.449260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.449233 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:31.449458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.449436 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:31.471705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.471672 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:31.472085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.472062 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:31.472255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.472229 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:31.506016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.505963 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:31.506355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:31.506311 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:32.309978 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:32.309907 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:29:32.405742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:32.404735 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:32.405742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:32.404819 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:32.452740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:32.447373 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:32.452740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:32.447442 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:32.461742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:32.460140 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:32.461742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:32.460195 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:33.462772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:33.450225 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:33.462772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:33.450288 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:33.688741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:33.687819 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:33.688741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:33.687892 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:33.688741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:33.688416 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:33.688741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:33.688469 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:34.448742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.441014 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:34.448742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.443244 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:34.474187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.474146 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:34.482990 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.482951 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:34.483232 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.475042 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:34.501027 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.500990 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:34.501282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.487205 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:34.501400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.501383 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:34.916831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:34.916779 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:35.461570 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:35.460900 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:35.461570 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:35.460977 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:35.510241 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:35.475347 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:35.511222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:35.511193 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:35.511417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:35.511391 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:35.583742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:35.530841 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:35.583742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:35.530909 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:36.404114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:36.404067 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:36.404404 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:36.404373 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:36.447745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:36.444273 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:36.448045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:36.448009 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:36.476263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:36.476228 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:36.476544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:36.476515 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:37.419023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.418988 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:37.419625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.419591 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:37.451746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.445262 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:37.457083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.457046 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:37.461097 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.461067 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:37.461276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.461252 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:37.844825 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.820904 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:29:37.888848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:37.874701 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podStartSLOduration=9.874624627 podStartE2EDuration="9.874624627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:31.175118783 +0000 UTC m=+992.604454817" watchObservedRunningTime="2026-11-03 22:29:37.874624627 +0000 UTC m=+999.304017079" Nov 03 22:29:38.400812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:38.400776 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:29:38.401641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:38.400842 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:29:38.447479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:38.446199 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:29:38.450884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:38.450840 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 22:29:38.459468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:38.459439 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:29:38.459677 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:38.459652 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 22:29:39.411733 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.408007 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:39.411733 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.409481 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:39.411733 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.410478 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:29:39.471556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.471515 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6"] Nov 03 22:29:39.494700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.487292 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.494700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.492866 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6"] Nov 03 22:29:39.520590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.517909 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-51bfa2e1-serving-cert\"" Nov 03 22:29:39.555793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.549755 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:39.555793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.551115 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:39.558328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.558259 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:29:39.590992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.590952 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-metrics-certs\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.591180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.591029 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-openshift-service-ca\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.591180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.591058 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-maas-api-tls\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.591295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.591242 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklcl\" (UniqueName: \"kubernetes.io/projected/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-kube-api-access-bklcl\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.629906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.629869 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:39.630650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.630437 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:39.631459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.631432 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:29:39.693117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.693029 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklcl\" (UniqueName: \"kubernetes.io/projected/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-kube-api-access-bklcl\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.693285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.693194 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-metrics-certs\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.693285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.693233 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-openshift-service-ca\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.693285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.693260 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-maas-api-tls\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.694188 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.694154 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-openshift-service-ca\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.696298 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.696268 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-maas-api-tls\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.696445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.696401 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-metrics-certs\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.728233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.728137 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklcl\" (UniqueName: \"kubernetes.io/projected/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-kube-api-access-bklcl\") pod \"maas-api-e2e-ait-51bfa2e1-8698958694-88fk6\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:39.870350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:39.870312 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:40.506202 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:40.504083 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6"] Nov 03 22:29:40.851056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:40.838532 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" event={"ID":"3f7dcea8-98ec-46c4-8a33-84abca5f6c96","Type":"ContainerStarted","Data":"8427ee9e7b7cfa3cf5d7d7175fb275db338b10dc3413c1edc34911da0655c92a"} Nov 03 22:29:41.846205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:41.846161 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" event={"ID":"3f7dcea8-98ec-46c4-8a33-84abca5f6c96","Type":"ContainerStarted","Data":"c485914ecc6afb976bcacbe25bf76f66b8a0dd36626999537fb01211c5d8b08c"} Nov 03 22:29:41.846695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:41.846271 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:41.887130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:41.887084 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" podStartSLOduration=2.192460564 podStartE2EDuration="2.887071482s" totalImagesPullingTime="694.610918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:41.885636317 +0000 UTC m=+1003.314972312" watchObservedRunningTime="2026-11-03 22:29:41.887071482 +0000 UTC m=+1003.316407474" Nov 03 22:29:44.507926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:44.507889 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4"] Nov 03 22:29:44.540303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:44.540273 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh"] Nov 03 22:29:48.034691 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.032642 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn"] Nov 03 22:29:48.041226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.041197 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.058227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.058188 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn"] Nov 03 22:29:48.058470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.058449 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-53879c92\"" Nov 03 22:29:48.097320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.097287 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z"] Nov 03 22:29:48.101993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.101972 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:48.120532 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.120504 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-53879c92-dockercfg-6h2pl\"" Nov 03 22:29:48.120878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.120857 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z"] Nov 03 22:29:48.129518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.129489 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd44272-3cab-46a1-92d0-e3a5849156d8-plugins-config-volume\") pod \"payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.129849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.129825 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glpln\" (UniqueName: \"kubernetes.io/projected/5fd44272-3cab-46a1-92d0-e3a5849156d8-kube-api-access-glpln\") pod \"payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.265741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.251537 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24k7\" (UniqueName: \"kubernetes.io/projected/3e6689e8-ec4d-4b52-a443-e882316b71f8-kube-api-access-s24k7\") pod \"payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:48.265741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.251804 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6689e8-ec4d-4b52-a443-e882316b71f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:48.265741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.251859 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glpln\" (UniqueName: \"kubernetes.io/projected/5fd44272-3cab-46a1-92d0-e3a5849156d8-kube-api-access-glpln\") pod \"payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.265741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.251978 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd44272-3cab-46a1-92d0-e3a5849156d8-plugins-config-volume\") pod \"payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.265741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.253125 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd44272-3cab-46a1-92d0-e3a5849156d8-plugins-config-volume\") pod \"payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.412655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.412623 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:29:48.414164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.414103 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24k7\" (UniqueName: \"kubernetes.io/projected/3e6689e8-ec4d-4b52-a443-e882316b71f8-kube-api-access-s24k7\") pod \"payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:48.415526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.414776 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6689e8-ec4d-4b52-a443-e882316b71f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:48.417897 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.417186 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6689e8-ec4d-4b52-a443-e882316b71f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:48.476751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.453475 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glpln\" (UniqueName: \"kubernetes.io/projected/5fd44272-3cab-46a1-92d0-e3a5849156d8-kube-api-access-glpln\") pod \"payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.537949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.537531 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24k7\" (UniqueName: \"kubernetes.io/projected/3e6689e8-ec4d-4b52-a443-e882316b71f8-kube-api-access-s24k7\") pod \"payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:48.687788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.680085 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:48.725944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:48.725903 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:49.509175 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:49.509066 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" podUID="bd419ba9-cecb-4d9c-9466-2d8d42245508" containerName="payload-processing" containerID="cri-o://43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f" gracePeriod=25 Nov 03 22:29:49.541581 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:49.541541 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" podUID="32665d6c-ab04-4675-81eb-4388bc4e6acc" containerName="payload-pre-processing" containerID="cri-o://792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11" gracePeriod=25 Nov 03 22:29:49.587059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:49.587034 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn"] Nov 03 22:29:49.594935 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:29:49.594895 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd44272_3cab_46a1_92d0_e3a5849156d8.slice/crio-17f00903c0315ac77a341c5647b52233f745ac925212a7bc1f194a9c035f6fbe WatchSource:0}: Error finding container 17f00903c0315ac77a341c5647b52233f745ac925212a7bc1f194a9c035f6fbe: Status 404 returned error can't find the container with id 17f00903c0315ac77a341c5647b52233f745ac925212a7bc1f194a9c035f6fbe Nov 03 22:29:49.610055 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:49.610031 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4"] Nov 03 22:29:49.634410 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:29:49.634383 4625 logging.go:55] [core] [Channel #461 SubChannel #462]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.92:50052", ServerName: "10.134.0.92:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.92:50052: connect: connection refused" Nov 03 22:29:49.748235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:49.748120 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z"] Nov 03 22:29:50.690132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:50.690022 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" podUID="bd419ba9-cecb-4d9c-9466-2d8d42245508" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.92:50052\" within 1s: context deadline exceeded" Nov 03 22:29:50.876594 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:50.876553 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:29:50.909738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:50.901827 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd419ba9-cecb-4d9c-9466-2d8d42245508-kube-api-access-97b4g\" (UniqueName: \"kubernetes.io/projected/bd419ba9-cecb-4d9c-9466-2d8d42245508-kube-api-access-97b4g\") pod \"bd419ba9-cecb-4d9c-9466-2d8d42245508\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " Nov 03 22:29:50.910090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:50.910056 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd419ba9-cecb-4d9c-9466-2d8d42245508-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419ba9-cecb-4d9c-9466-2d8d42245508-plugins-config-volume\") pod \"bd419ba9-cecb-4d9c-9466-2d8d42245508\" (UID: \"bd419ba9-cecb-4d9c-9466-2d8d42245508\") " Nov 03 22:29:51.021888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.018446 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd419ba9-cecb-4d9c-9466-2d8d42245508-plugins-config-volume" pod "bd419ba9-cecb-4d9c-9466-2d8d42245508" (UID: "bd419ba9-cecb-4d9c-9466-2d8d42245508"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:51.242742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.210909 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419ba9-cecb-4d9c-9466-2d8d42245508-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:29:51.448738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.445964 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:51.448738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.447633 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:51.542379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.542331 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd419ba9-cecb-4d9c-9466-2d8d42245508-kube-api-access-97b4g" pod "bd419ba9-cecb-4d9c-9466-2d8d42245508" (UID: "bd419ba9-cecb-4d9c-9466-2d8d42245508"). InnerVolumeSpecName "kube-api-access-97b4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:51.580252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.580103 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:51.707785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.672160 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97b4g\" (UniqueName: \"kubernetes.io/projected/bd419ba9-cecb-4d9c-9466-2d8d42245508-kube-api-access-97b4g\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:29:51.757421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.735019 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:29:51.814742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.776331 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" event={"ID":"3e6689e8-ec4d-4b52-a443-e882316b71f8","Type":"ContainerStarted","Data":"a0011a9d2da2aae49aca2de26befd4d0ca3e411af5b9c157830b9bd4018d6582"} Nov 03 22:29:51.880597 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.880559 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32665d6c-ab04-4675-81eb-4388bc4e6acc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32665d6c-ab04-4675-81eb-4388bc4e6acc-plugins-config-volume\") pod \"32665d6c-ab04-4675-81eb-4388bc4e6acc\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " Nov 03 22:29:51.881087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.881066 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32665d6c-ab04-4675-81eb-4388bc4e6acc-kube-api-access-8m842\" (UniqueName: \"kubernetes.io/projected/32665d6c-ab04-4675-81eb-4388bc4e6acc-kube-api-access-8m842\") pod \"32665d6c-ab04-4675-81eb-4388bc4e6acc\" (UID: \"32665d6c-ab04-4675-81eb-4388bc4e6acc\") " Nov 03 22:29:51.927189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.927136 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32665d6c-ab04-4675-81eb-4388bc4e6acc-plugins-config-volume" pod "32665d6c-ab04-4675-81eb-4388bc4e6acc" (UID: "32665d6c-ab04-4675-81eb-4388bc4e6acc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:51.936780 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:51.933395 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32665d6c-ab04-4675-81eb-4388bc4e6acc-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:29:52.211681 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.211640 4625 generic.go:356] "Generic (PLEG): container finished" podID="32665d6c-ab04-4675-81eb-4388bc4e6acc" containerID="792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11" exitCode=0 Nov 03 22:29:52.212153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.212122 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" event={"ID":"32665d6c-ab04-4675-81eb-4388bc4e6acc","Type":"ContainerDied","Data":"792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11"} Nov 03 22:29:52.212350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.212329 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" event={"ID":"32665d6c-ab04-4675-81eb-4388bc4e6acc","Type":"ContainerDied","Data":"fba4d14804505cc34cd8da633e44acb4f801f3f91d229524fe5cbaab43168f5a"} Nov 03 22:29:52.212487 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.212473 4625 scope.go:122] "RemoveContainer" containerID="792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11" Nov 03 22:29:52.229796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.214520 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh" Nov 03 22:29:52.260047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.257166 4625 generic.go:356] "Generic (PLEG): container finished" podID="bd419ba9-cecb-4d9c-9466-2d8d42245508" containerID="43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f" exitCode=0 Nov 03 22:29:52.260362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.260334 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" event={"ID":"bd419ba9-cecb-4d9c-9466-2d8d42245508","Type":"ContainerDied","Data":"43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f"} Nov 03 22:29:52.260503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.260490 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" event={"ID":"bd419ba9-cecb-4d9c-9466-2d8d42245508","Type":"ContainerDied","Data":"5830ee6e3e1282fcf04936ab17f7d400c4277e5ad13dc10128d0f7f965015824"} Nov 03 22:29:52.280740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.260641 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4" Nov 03 22:29:52.284221 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.283675 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd"] Nov 03 22:29:52.325290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.304962 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" podUID="ee156b68-b18c-48a0-ba5e-5ddac259c3b9" containerName="istio-proxy" containerID="cri-o://18bdf1d1f73698d5395fba91b717f7c54d1336b39589fd62f61c5f090e63b5cb" gracePeriod=30 Nov 03 22:29:52.428081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.427981 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" event={"ID":"5fd44272-3cab-46a1-92d0-e3a5849156d8","Type":"ContainerStarted","Data":"4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5"} Nov 03 22:29:52.432746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.431060 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:52.432746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.431102 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" event={"ID":"5fd44272-3cab-46a1-92d0-e3a5849156d8","Type":"ContainerStarted","Data":"17f00903c0315ac77a341c5647b52233f745ac925212a7bc1f194a9c035f6fbe"} Nov 03 22:29:52.460741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.453081 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:52.460741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.454072 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:29:52.470894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.467875 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:52.500747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.497848 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:52.522743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.521295 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:52.522743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.521819 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:29:52.678788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.677407 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32665d6c-ab04-4675-81eb-4388bc4e6acc-kube-api-access-8m842" pod "32665d6c-ab04-4675-81eb-4388bc4e6acc" (UID: "32665d6c-ab04-4675-81eb-4388bc4e6acc"). InnerVolumeSpecName "kube-api-access-8m842". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:52.678788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.677576 4625 scope.go:122] "RemoveContainer" containerID="792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11" Nov 03 22:29:52.696745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.693885 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:52.696745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.694393 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:29:52.722863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.720702 4625 patch_prober.go:29] interesting pod/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:29:52.722863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.720808 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" podUID="ee156b68-b18c-48a0-ba5e-5ddac259c3b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:29:52.787335 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.787304 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m842\" (UniqueName: \"kubernetes.io/projected/32665d6c-ab04-4675-81eb-4388bc4e6acc-kube-api-access-8m842\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:29:52.813933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.813889 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:52.814208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.814179 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:52.890084 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:52.889779 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11\": container with ID starting with 792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11 not found: ID does not exist" containerID="792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11" Nov 03 22:29:52.890084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.889831 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11"} err="failed to get container status \"792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11\": rpc error: code = NotFound desc = could not find container \"792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11\": container with ID starting with 792dd74810b3cc0fb5db85a1b7958565d706ed2171cf58404a88263ded4e9f11 not found: ID does not exist" Nov 03 22:29:52.890084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:52.889859 4625 scope.go:122] "RemoveContainer" containerID="43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f" Nov 03 22:29:53.123793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.113943 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4"] Nov 03 22:29:53.561742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.516238 4625 scope.go:122] "RemoveContainer" containerID="43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f" Nov 03 22:29:53.561742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.516333 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:29:53.561742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.518260 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:29:53.561742 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:53.520665 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f\": container with ID starting with 43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f not found: ID does not exist" containerID="43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f" Nov 03 22:29:53.561742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.520698 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f"} err="failed to get container status \"43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f\": rpc error: code = NotFound desc = could not find container \"43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f\": container with ID starting with 43fa4633bea0358dbbca4a75519949932c896e491a52b994a80c313b2034107f not found: ID does not exist" Nov 03 22:29:53.561742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.534599 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:53.561742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.543702 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:53.581147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.581108 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7b10505-5d8bb99fb-q2vm4"] Nov 03 22:29:53.712860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.712815 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:53.757954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:53.757801 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" podStartSLOduration=5.757779274 podStartE2EDuration="5.757779274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:52.813917027 +0000 UTC m=+1014.243253021" watchObservedRunningTime="2026-11-03 22:29:53.757779274 +0000 UTC m=+1015.187115268" Nov 03 22:29:54.181269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.181149 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" event={"ID":"3e6689e8-ec4d-4b52-a443-e882316b71f8","Type":"ContainerStarted","Data":"92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8"} Nov 03 22:29:54.197965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.197929 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:54.447459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.447357 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:29:54.448114 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:54.448080 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32665d6c_ab04_4675_81eb_4388bc4e6acc.slice/crio-fba4d14804505cc34cd8da633e44acb4f801f3f91d229524fe5cbaab43168f5a\": RecentStats: unable to find data in memory cache]" Nov 03 22:29:54.532506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.532113 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:54.536841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.536801 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:54.617588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.617398 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:54.754995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.754893 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:54.775737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.769732 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 03 22:29:54.788740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.777010 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" podUID="84f0bb48-07ef-4193-80ae-004397ef48b7" containerName="payload-processing" containerID="cri-o://74b71df6fc80cb9cc6d54c4dd46a06d2a599a0e2d8e2a9187dd218c392376949" gracePeriod=25 Nov 03 22:29:55.125748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:54.937768 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Nov 03 22:29:55.576758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.569451 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:55.581749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.580780 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.611742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.583094 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwbc8" podUID="f072ea9a-f752-461e-85e1-3297b846fde0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.626538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.626206 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.790930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.790382 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.815786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.815380 4625 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:55.815786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.815461 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.026760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.912308 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:56.026760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:55.912387 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.268755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:56.257546 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd419ba9-cecb-4d9c-9466-2d8d42245508" path="/var/lib/kubelet/pods/bd419ba9-cecb-4d9c-9466-2d8d42245508/volumes" Nov 03 22:29:56.303009 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:29:56.302928 4625 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.583s" Nov 03 22:29:56.573496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:56.573398 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.578629 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:56.576522 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:29:56.583264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:56.583233 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.633289 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:56.632783 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.816741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:56.811866 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.084742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.079506 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh"] Nov 03 22:29:57.213679 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.208791 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b7b10505-775cb59b75-96cjh"] Nov 03 22:29:57.331752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.314429 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" podStartSLOduration=9.314408396 podStartE2EDuration="9.314408396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:54.238049339 +0000 UTC m=+1015.667385323" watchObservedRunningTime="2026-11-03 22:29:57.314408396 +0000 UTC m=+1018.743744402" Nov 03 22:29:57.579743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.579689 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.648555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.645636 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.680808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.675285 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.732274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.719710 4625 generic.go:356] "Generic (PLEG): container finished" podID="84f0bb48-07ef-4193-80ae-004397ef48b7" containerID="74b71df6fc80cb9cc6d54c4dd46a06d2a599a0e2d8e2a9187dd218c392376949" exitCode=0 Nov 03 22:29:57.736870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.732887 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" event={"ID":"84f0bb48-07ef-4193-80ae-004397ef48b7","Type":"ContainerDied","Data":"74b71df6fc80cb9cc6d54c4dd46a06d2a599a0e2d8e2a9187dd218c392376949"} Nov 03 22:29:57.824945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:57.821810 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:58.601768 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:58.597126 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.626179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:58.625599 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.663758 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:29:58.659405 4625 logging.go:55] [core] [Channel #480 SubChannel #481]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.96:50052", ServerName: "10.134.0.96:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.96:50052: connect: connection refused" Nov 03 22:29:58.663758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:58.662658 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.732869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:58.684809 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:29:58.937855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:58.936243 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:29:59.250886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.250327 4625 generic.go:356] "Generic (PLEG): container finished" podID="ee156b68-b18c-48a0-ba5e-5ddac259c3b9" containerID="18bdf1d1f73698d5395fba91b717f7c54d1336b39589fd62f61c5f090e63b5cb" exitCode=0 Nov 03 22:29:59.607740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.607336 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" podUID="84f0bb48-07ef-4193-80ae-004397ef48b7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.96:50052\" within 1s: context deadline exceeded" Nov 03 22:29:59.607740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.607680 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:29:59.695743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.689844 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.695743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.691289 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.695999 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.695968 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32665d6c-ab04-4675-81eb-4388bc4e6acc" path="/var/lib/kubelet/pods/32665d6c-ab04-4675-81eb-4388bc4e6acc/volumes" Nov 03 22:29:59.722773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.696483 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" event={"ID":"ee156b68-b18c-48a0-ba5e-5ddac259c3b9","Type":"ContainerDied","Data":"18bdf1d1f73698d5395fba91b717f7c54d1336b39589fd62f61c5f090e63b5cb"} Nov 03 22:29:59.722773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.696556 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm"] Nov 03 22:29:59.722773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.696603 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88"] Nov 03 22:29:59.724908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.724865 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" podUID="03a8793d-65b1-4529-acfd-c3ae76f32ba0" containerName="maas-api" containerID="cri-o://57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba" gracePeriod=30 Nov 03 22:29:59.831233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.818540 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" podUID="b963989a-255c-41b8-84a3-f3ffc60141ac" containerName="istio-proxy" containerID="cri-o://163585c2a9aa501781237492aa63e5c3d02118c4517d9dacd820a62d97768de9" gracePeriod=30 Nov 03 22:29:59.946086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:29:59.937060 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:30:00.856650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:00.856598 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:00.859580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:00.859540 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:30:01.166805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:01.165244 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:02.066560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:01.966174 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" podUID="6f8dc406-d4b0-407a-bd99-069aa88016cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:02.149151 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.148115 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:02.160748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.150761 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:02.210767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.165920 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:02.758798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.758288 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Nov 03 22:30:02.797741 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:02.760473 4625 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.986s" Nov 03 22:30:02.797741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.760535 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g"] Nov 03 22:30:02.797741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.761648 4625 cpu_manager.go:414] "containerMap: removing container" podUID="bd419ba9-cecb-4d9c-9466-2d8d42245508" containerName="payload-processing" Nov 03 22:30:02.797741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.761668 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd419ba9-cecb-4d9c-9466-2d8d42245508" containerName="payload-processing" Nov 03 22:30:02.842951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:02.842401 4625 generic.go:356] "Generic (PLEG): container finished" podID="03a8793d-65b1-4529-acfd-c3ae76f32ba0" containerID="57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba" exitCode=0 Nov 03 22:30:03.124181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.124136 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:30:03.147758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.139018 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:30:03.148840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.148792 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:03.212777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.182918 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:30:03.212777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.205781 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:03.227872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.227828 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:30:03.267483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.267430 4625 cpu_manager.go:414] "containerMap: removing container" podUID="32665d6c-ab04-4675-81eb-4388bc4e6acc" containerName="payload-pre-processing" Nov 03 22:30:03.267483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.267457 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32665d6c-ab04-4675-81eb-4388bc4e6acc" containerName="payload-pre-processing" Nov 03 22:30:03.283889 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.283851 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="32665d6c-ab04-4675-81eb-4388bc4e6acc" containerName="payload-pre-processing" Nov 03 22:30:03.284132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.284114 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f0bb48-07ef-4193-80ae-004397ef48b7" containerName="payload-processing" Nov 03 22:30:03.284276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.284263 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd419ba9-cecb-4d9c-9466-2d8d42245508" containerName="payload-processing" Nov 03 22:30:03.284417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.284403 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee156b68-b18c-48a0-ba5e-5ddac259c3b9" containerName="istio-proxy" Nov 03 22:30:03.310692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.310655 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-kube-api-access-mdk6n\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-kube-api-access-mdk6n\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.311017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.310993 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-envoy\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.311194 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.311176 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-credential-socket\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.311319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.311301 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-data\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.311527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.311506 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istiod-ca-cert\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.311673 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.311657 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f0bb48-07ef-4193-80ae-004397ef48b7-kube-api-access-vl2ql\" (UniqueName: \"kubernetes.io/projected/84f0bb48-07ef-4193-80ae-004397ef48b7-kube-api-access-vl2ql\") pod \"84f0bb48-07ef-4193-80ae-004397ef48b7\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " Nov 03 22:30:03.311822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.311805 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-socket\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.311986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.311970 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-certs\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.312168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.312152 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-podinfo\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.312361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.312344 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f0bb48-07ef-4193-80ae-004397ef48b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f0bb48-07ef-4193-80ae-004397ef48b7-plugins-config-volume\") pod \"84f0bb48-07ef-4193-80ae-004397ef48b7\" (UID: \"84f0bb48-07ef-4193-80ae-004397ef48b7\") " Nov 03 22:30:03.312519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.312503 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-token\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-token\") pod \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\" (UID: \"ee156b68-b18c-48a0-ba5e-5ddac259c3b9\") " Nov 03 22:30:03.314964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.314939 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-socket" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:03.315428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.315406 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-certs" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:03.315965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.315941 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istiod-ca-cert" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:03.316412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.316391 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-credential-socket" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:03.316750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.316730 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-data" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:03.486803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.389038 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-kube-api-access-mdk6n" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "kube-api-access-mdk6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:03.521743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.411023 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f0bb48-07ef-4193-80ae-004397ef48b7-plugins-config-volume" pod "84f0bb48-07ef-4193-80ae-004397ef48b7" (UID: "84f0bb48-07ef-4193-80ae-004397ef48b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:03.521743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.452836 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.521743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.513546 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.521743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.513568 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.521743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.513583 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.521743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.513598 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.521743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.508905 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:03.644103 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:03.428956 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:30:03.644103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.581525 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-podinfo" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:30:03.644103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.606343 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f0bb48-07ef-4193-80ae-004397ef48b7-kube-api-access-vl2ql" pod "84f0bb48-07ef-4193-80ae-004397ef48b7" (UID: "84f0bb48-07ef-4193-80ae-004397ef48b7"). InnerVolumeSpecName "kube-api-access-vl2ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:03.644103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.632392 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f0bb48-07ef-4193-80ae-004397ef48b7-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.644103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.632431 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdk6n\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-kube-api-access-mdk6n\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.644103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.632448 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2ql\" (UniqueName: \"kubernetes.io/projected/84f0bb48-07ef-4193-80ae-004397ef48b7-kube-api-access-vl2ql\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.644103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.632462 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:03.731453 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.731416 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-envoy" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:04.526550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.380155 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-token" pod "ee156b68-b18c-48a0-ba5e-5ddac259c3b9" (UID: "ee156b68-b18c-48a0-ba5e-5ddac259c3b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:04.527127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.134235 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:04.549651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.154837 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:04.549651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.245205 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:04.549651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.287823 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:04.549651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.330596 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzs7v\" (UniqueName: \"kubernetes.io/projected/15f72d4d-599c-49ab-93fb-7b566b020708-kube-api-access-fzs7v\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:04.568787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.444762 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" event={"ID":"03a8793d-65b1-4529-acfd-c3ae76f32ba0","Type":"ContainerDied","Data":"57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba"} Nov 03 22:30:04.569952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:03.923344 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:30:04.570273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.569394 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f72d4d-599c-49ab-93fb-7b566b020708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:04.570756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.570738 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:04.570890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.570878 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee156b68-b18c-48a0-ba5e-5ddac259c3b9-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:04.571377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.571359 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.778866 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4"] Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.780012 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ee156b68-b18c-48a0-ba5e-5ddac259c3b9" containerName="istio-proxy" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.780032 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee156b68-b18c-48a0-ba5e-5ddac259c3b9" containerName="istio-proxy" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.780070 4625 cpu_manager.go:414] "containerMap: removing container" podUID="84f0bb48-07ef-4193-80ae-004397ef48b7" containerName="payload-processing" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.780078 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f0bb48-07ef-4193-80ae-004397ef48b7" containerName="payload-processing" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.725233 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzs7v\" (UniqueName: \"kubernetes.io/projected/15f72d4d-599c-49ab-93fb-7b566b020708-kube-api-access-fzs7v\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.785552 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.785647 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f72d4d-599c-49ab-93fb-7b566b020708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:04.787778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.786823 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f72d4d-599c-49ab-93fb-7b566b020708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:04.816885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.798987 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzs7v\" (UniqueName: \"kubernetes.io/projected/15f72d4d-599c-49ab-93fb-7b566b020708-kube-api-access-fzs7v\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:04.852983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.852945 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:04.853811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:04.853785 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" Nov 03 22:30:05.117834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.084202 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" Nov 03 22:30:05.117834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.097906 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:30:05.117834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.097994 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:30:05.117834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.098016 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk"] Nov 03 22:30:05.117834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.114105 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:05.245213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.245183 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:30:05.299191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.299160 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4"] Nov 03 22:30:05.354548 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.354520 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g"] Nov 03 22:30:05.354925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.354898 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4" event={"ID":"84f0bb48-07ef-4193-80ae-004397ef48b7","Type":"ContainerDied","Data":"d9acd333311f724c7b969939dbbaabdc01298f5f2c54ad698c63d958a55a4041"} Nov 03 22:30:05.374219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.373977 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd" event={"ID":"ee156b68-b18c-48a0-ba5e-5ddac259c3b9","Type":"ContainerDied","Data":"1a8f0887c7f2fc68fb7254e8e56b69c1750e84ff193e0d0666d3308dc4c941b7"} Nov 03 22:30:05.391908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.391874 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk"] Nov 03 22:30:05.394765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.349694 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a8793d-65b1-4529-acfd-c3ae76f32ba0-kube-api-access-8wk56\" (UniqueName: \"kubernetes.io/projected/03a8793d-65b1-4529-acfd-c3ae76f32ba0-kube-api-access-8wk56\") pod \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " Nov 03 22:30:05.401771 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.395082 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\") pod \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " Nov 03 22:30:05.418922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.411449 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-metrics-certs\") pod \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " Nov 03 22:30:05.545773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.520596 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a8793d-65b1-4529-acfd-c3ae76f32ba0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a8793d-65b1-4529-acfd-c3ae76f32ba0-openshift-service-ca\") pod \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\" (UID: \"03a8793d-65b1-4529-acfd-c3ae76f32ba0\") " Nov 03 22:30:05.796967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.796927 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8934af8-8db7-4ce4-a899-7abdfef8a34f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:05.797227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.797204 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e87482-394e-481d-993b-590fd0a6eb3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:05.797622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.797602 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98m5p\" (UniqueName: \"kubernetes.io/projected/a7e87482-394e-481d-993b-590fd0a6eb3e-kube-api-access-98m5p\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:05.797871 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.797852 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2kzn\" (UniqueName: \"kubernetes.io/projected/c8934af8-8db7-4ce4-a899-7abdfef8a34f-kube-api-access-l2kzn\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:05.798139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.564780 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:30:05.798577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.299446 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:05.799438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.533127 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a8793d-65b1-4529-acfd-c3ae76f32ba0-openshift-service-ca" pod "03a8793d-65b1-4529-acfd-c3ae76f32ba0" (UID: "03a8793d-65b1-4529-acfd-c3ae76f32ba0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:05.799569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.592955 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls" pod "03a8793d-65b1-4529-acfd-c3ae76f32ba0" (UID: "03a8793d-65b1-4529-acfd-c3ae76f32ba0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:05.799684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.374049 4625 scope.go:122] "RemoveContainer" containerID="74b71df6fc80cb9cc6d54c4dd46a06d2a599a0e2d8e2a9187dd218c392376949" Nov 03 22:30:05.800574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.555199 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:05.801204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.762316 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:30:05.801356 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.801341 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74"] Nov 03 22:30:05.802588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.802566 4625 cpu_manager.go:414] "containerMap: removing container" podUID="03a8793d-65b1-4529-acfd-c3ae76f32ba0" containerName="maas-api" Nov 03 22:30:05.802741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.802705 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a8793d-65b1-4529-acfd-c3ae76f32ba0" containerName="maas-api" Nov 03 22:30:05.803089 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.803072 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a8793d-65b1-4529-acfd-c3ae76f32ba0" containerName="maas-api" Nov 03 22:30:05.808918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:05.808887 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-metrics-certs" pod "03a8793d-65b1-4529-acfd-c3ae76f32ba0" (UID: "03a8793d-65b1-4529-acfd-c3ae76f32ba0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:06.022659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.016203 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74"] Nov 03 22:30:06.261803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.221668 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a8793d-65b1-4529-acfd-c3ae76f32ba0-kube-api-access-8wk56" pod "03a8793d-65b1-4529-acfd-c3ae76f32ba0" (UID: "03a8793d-65b1-4529-acfd-c3ae76f32ba0"). InnerVolumeSpecName "kube-api-access-8wk56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:06.296740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.266190 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.589580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.272136 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4"] Nov 03 22:30:06.589580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.588467 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b279244-855ccd57cb-l9wh4"] Nov 03 22:30:06.589580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.399157 4625 scope.go:122] "RemoveContainer" containerID="18bdf1d1f73698d5395fba91b717f7c54d1336b39589fd62f61c5f090e63b5cb" Nov 03 22:30:06.788253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.788219 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-podinfo\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.788608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.788589 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff85a6f-57be-4d67-b5f8-a6a630d33391-istiod-ca-cert\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.788804 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.788786 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98m5p\" (UniqueName: \"kubernetes.io/projected/a7e87482-394e-481d-993b-590fd0a6eb3e-kube-api-access-98m5p\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:06.788956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.788939 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-socket\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.789167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.789150 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-envoy\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.789343 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.789325 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2kzn\" (UniqueName: \"kubernetes.io/projected/c8934af8-8db7-4ce4-a899-7abdfef8a34f-kube-api-access-l2kzn\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:06.789473 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.789460 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-certs\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.789611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.789597 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27g4\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-kube-api-access-x27g4\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.789844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.789827 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-token\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.790032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.790015 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8934af8-8db7-4ce4-a899-7abdfef8a34f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:06.790430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.790411 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-data\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.790592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.790573 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e87482-394e-481d-993b-590fd0a6eb3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:06.790769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.790753 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-credential-socket\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:06.791474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.791454 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:06.791603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.791586 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a8793d-65b1-4529-acfd-c3ae76f32ba0-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:06.791707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.791695 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wk56\" (UniqueName: \"kubernetes.io/projected/03a8793d-65b1-4529-acfd-c3ae76f32ba0-kube-api-access-8wk56\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:06.791824 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.791810 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a8793d-65b1-4529-acfd-c3ae76f32ba0-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:06.812873 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:06.812117 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:07.203739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.201917 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8934af8-8db7-4ce4-a899-7abdfef8a34f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:07.203739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.202164 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e87482-394e-481d-993b-590fd0a6eb3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:07.235479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.226942 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:07.235479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.229775 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:07.415030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.414983 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" podUID="b4d2e243-3b00-4807-8324-e14cb06f0d20" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.516767 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-token\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.709401 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-data\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.709529 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-credential-socket\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.709680 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-podinfo\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.709861 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff85a6f-57be-4d67-b5f8-a6a630d33391-istiod-ca-cert\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.709958 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-socket\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.710065 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-envoy\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.710121 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-certs\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.710189 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27g4\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-kube-api-access-x27g4\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.527390 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cbe6e297-openshift-default-dockercfg-p5g9h\"" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.711686 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-data\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.625730 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" podUID="3e6689e8-ec4d-4b52-a443-e882316b71f8" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 03 22:30:07.717159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.712337 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-credential-socket\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.796927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.736429 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff85a6f-57be-4d67-b5f8-a6a630d33391-istiod-ca-cert\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.796927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.736761 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-socket\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:07.796927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.785658 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" podUID="03a8793d-65b1-4529-acfd-c3ae76f32ba0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.98:8443/health\": context deadline exceeded" Nov 03 22:30:07.961740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:07.940873 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-certs\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:08.000446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.000348 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f0bb48-07ef-4193-80ae-004397ef48b7" path="/var/lib/kubelet/pods/84f0bb48-07ef-4193-80ae-004397ef48b7/volumes" Nov 03 22:30:08.014756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.014441 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2kzn\" (UniqueName: \"kubernetes.io/projected/c8934af8-8db7-4ce4-a899-7abdfef8a34f-kube-api-access-l2kzn\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:08.014756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.014545 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:08.020157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.019960 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:08.025204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.024478 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-podinfo\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:08.068072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.068028 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd"] Nov 03 22:30:08.069462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.069428 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b7b10505-gw-openshift-default-5778588977-67psd"] Nov 03 22:30:08.086381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.085737 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27g4\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-kube-api-access-x27g4\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:08.105535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.105502 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98m5p\" (UniqueName: \"kubernetes.io/projected/a7e87482-394e-481d-993b-590fd0a6eb3e-kube-api-access-98m5p\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:08.106497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.106474 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-envoy\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:08.112555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.112530 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-token\") pod \"e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:08.128816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.128790 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g"] Nov 03 22:30:08.195749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.192995 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:30:08.195749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.193062 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:30:08.199627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.196286 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" event={"ID":"03a8793d-65b1-4529-acfd-c3ae76f32ba0","Type":"ContainerDied","Data":"34df7dd14b39d25a89be58b1173e364add9c0e172e01f4d2a127b4dc8c23c40b"} Nov 03 22:30:08.199627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.196361 4625 scope.go:122] "RemoveContainer" containerID="57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba" Nov 03 22:30:08.199627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.196908 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm" Nov 03 22:30:08.279275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.279027 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:08.314612 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:08.313187 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-34df7dd14b39d25a89be58b1173e364add9c0e172e01f4d2a127b4dc8c23c40b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:08.378054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.377794 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:08.401563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.401533 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc"] Nov 03 22:30:08.552915 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.549339 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.592797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.591475 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.592797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.591607 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbf2c\" (UniqueName: \"kubernetes.io/projected/c7873daa-811c-4d06-8d90-1bc378f4b149-kube-api-access-fbf2c\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.592797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.591663 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.679880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.674841 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm"] Nov 03 22:30:08.703198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.703112 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b279244-589ccd554-dvjtm"] Nov 03 22:30:08.722537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.704614 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.722537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.704728 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbf2c\" (UniqueName: \"kubernetes.io/projected/c7873daa-811c-4d06-8d90-1bc378f4b149-kube-api-access-fbf2c\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.722537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.704762 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.722537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.705910 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.722537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.708638 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:30:08.809632 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:08.809541 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7873daa_811c_4d06_8d90_1bc378f4b149.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:08.887194 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.813109 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbf2c\" (UniqueName: \"kubernetes.io/projected/c7873daa-811c-4d06-8d90-1bc378f4b149-kube-api-access-fbf2c\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.887194 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.819048 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:08.910674 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.910641 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a8793d-65b1-4529-acfd-c3ae76f32ba0" path="/var/lib/kubelet/pods/03a8793d-65b1-4529-acfd-c3ae76f32ba0/volumes" Nov 03 22:30:08.911660 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.911637 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee156b68-b18c-48a0-ba5e-5ddac259c3b9" path="/var/lib/kubelet/pods/ee156b68-b18c-48a0-ba5e-5ddac259c3b9/volumes" Nov 03 22:30:08.912261 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.912242 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc"] Nov 03 22:30:08.912443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.912429 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv"] Nov 03 22:30:08.927364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.927336 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv"] Nov 03 22:30:08.927628 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.927614 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:08.989742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:08.961936 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-53879c92-serving-cert\"" Nov 03 22:30:09.001349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.001319 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4"] Nov 03 22:30:09.063858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.045469 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvlr\" (UniqueName: \"kubernetes.io/projected/5f24a2a5-4268-4bd4-b505-b5aed592b177-kube-api-access-bqvlr\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.064169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.064137 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-metrics-certs\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.072906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.072860 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f24a2a5-4268-4bd4-b505-b5aed592b177-openshift-service-ca\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.096742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.073311 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-maas-api-tls\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.140850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.136029 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:09.288940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.276546 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvlr\" (UniqueName: \"kubernetes.io/projected/5f24a2a5-4268-4bd4-b505-b5aed592b177-kube-api-access-bqvlr\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.288940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.276636 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-metrics-certs\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.288940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.276742 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f24a2a5-4268-4bd4-b505-b5aed592b177-openshift-service-ca\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.288940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.276968 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-maas-api-tls\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.330446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.319672 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-maas-api-tls\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.355136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.336760 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk"] Nov 03 22:30:09.355136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.340130 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f24a2a5-4268-4bd4-b505-b5aed592b177-openshift-service-ca\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.389778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.377414 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74"] Nov 03 22:30:09.399771 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.399747 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-metrics-certs\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.406038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.402144 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvlr\" (UniqueName: \"kubernetes.io/projected/5f24a2a5-4268-4bd4-b505-b5aed592b177-kube-api-access-bqvlr\") pod \"maas-api-e2e-mt-53879c92-64786dd47b-z27zv\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.419564 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.419466 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerStarted","Data":"e3f231a934bfedb0c99db2750b9882696bb8c2f0c076d0ed7d9400b5aab29c19"} Nov 03 22:30:09.419564 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.419532 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerStarted","Data":"46613e7aec06a89b432eed6f24746baa2e3d259449da69a3f949cc3400a071c8"} Nov 03 22:30:09.425911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.424575 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:09.425911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.424649 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:09.425911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.424678 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:09.480240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.466597 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" podStartSLOduration=9.466576504 podStartE2EDuration="9.466576504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:09.461498167 +0000 UTC m=+1030.890834161" watchObservedRunningTime="2026-11-03 22:30:09.466576504 +0000 UTC m=+1030.895912498" Nov 03 22:30:09.683215 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.683174 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:09.784756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:09.779486 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc"] Nov 03 22:30:09.790914 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:30:09.790878 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7873daa_811c_4d06_8d90_1bc378f4b149.slice/crio-4974f3918b9dc0ffc41f9356d5255c4ffc997d5272f0e00832770633c58e533c WatchSource:0}: Error finding container 4974f3918b9dc0ffc41f9356d5255c4ffc997d5272f0e00832770633c58e533c: Status 404 returned error can't find the container with id 4974f3918b9dc0ffc41f9356d5255c4ffc997d5272f0e00832770633c58e533c Nov 03 22:30:10.177253 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:30:10.177207 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f24a2a5_4268_4bd4_b505_b5aed592b177.slice/crio-c6b42aafd383675b940c7263318e3550e7bee7f0237675cb49e123fe09399902 WatchSource:0}: Error finding container c6b42aafd383675b940c7263318e3550e7bee7f0237675cb49e123fe09399902: Status 404 returned error can't find the container with id c6b42aafd383675b940c7263318e3550e7bee7f0237675cb49e123fe09399902 Nov 03 22:30:10.250349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.250312 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv"] Nov 03 22:30:10.503201 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.503167 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" event={"ID":"c7873daa-811c-4d06-8d90-1bc378f4b149","Type":"ContainerStarted","Data":"4974f3918b9dc0ffc41f9356d5255c4ffc997d5272f0e00832770633c58e533c"} Nov 03 22:30:10.507202 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.507165 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerStarted","Data":"0a02dd42a353dfe9049a7bc78d5375781482523763846b9b8b718d79b9c219a3"} Nov 03 22:30:10.507336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.507224 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerStarted","Data":"e4e60dea1a3547f96bce561ade9256f625b48f8ddcf52e088f98785f2a6a2d74"} Nov 03 22:30:10.509206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.509180 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" event={"ID":"bff85a6f-57be-4d67-b5f8-a6a630d33391","Type":"ContainerStarted","Data":"4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b"} Nov 03 22:30:10.509292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.509227 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" event={"ID":"bff85a6f-57be-4d67-b5f8-a6a630d33391","Type":"ContainerStarted","Data":"120ccd2b8d218c1e468fb57ecb096b53d848201a0a6d8ba14e9acb0e5877dba0"} Nov 03 22:30:10.512272 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.512250 4625 generic.go:356] "Generic (PLEG): container finished" podID="15f72d4d-599c-49ab-93fb-7b566b020708" containerID="e3f231a934bfedb0c99db2750b9882696bb8c2f0c076d0ed7d9400b5aab29c19" exitCode=60 Nov 03 22:30:10.512365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.512328 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerDied","Data":"e3f231a934bfedb0c99db2750b9882696bb8c2f0c076d0ed7d9400b5aab29c19"} Nov 03 22:30:10.512592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.512575 4625 scope.go:122] "RemoveContainer" containerID="e3f231a934bfedb0c99db2750b9882696bb8c2f0c076d0ed7d9400b5aab29c19" Nov 03 22:30:10.514234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.514211 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerStarted","Data":"7a842635519a00a1b551b0f352d49cf853404b7ef5f9a27277d2c4b935299fc9"} Nov 03 22:30:10.514301 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.514257 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerStarted","Data":"0c263e4d44400176f0a18b1f76a19f72747506061b15c214acdc3c94172b0c7f"} Nov 03 22:30:10.520614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.520593 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" event={"ID":"5f24a2a5-4268-4bd4-b505-b5aed592b177","Type":"ContainerStarted","Data":"c6b42aafd383675b940c7263318e3550e7bee7f0237675cb49e123fe09399902"} Nov 03 22:30:10.549404 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.549356 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" podStartSLOduration=2.549343651 podStartE2EDuration="2.549343651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:10.544037652 +0000 UTC m=+1031.973373646" watchObservedRunningTime="2026-11-03 22:30:10.549343651 +0000 UTC m=+1031.978679653" Nov 03 22:30:10.591969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.591908 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" podStartSLOduration=10.59188925 podStartE2EDuration="10.59188925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:10.585835706 +0000 UTC m=+1032.015171699" watchObservedRunningTime="2026-11-03 22:30:10.59188925 +0000 UTC m=+1032.021225245" Nov 03 22:30:10.631184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.631128 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" podStartSLOduration=10.6311144 podStartE2EDuration="10.6311144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:10.629617483 +0000 UTC m=+1032.058953476" watchObservedRunningTime="2026-11-03 22:30:10.6311144 +0000 UTC m=+1032.060450393" Nov 03 22:30:10.708742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:10.708622 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" podStartSLOduration=10.708608394 podStartE2EDuration="10.708608394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:10.704264424 +0000 UTC m=+1032.133600417" watchObservedRunningTime="2026-11-03 22:30:10.708608394 +0000 UTC m=+1032.137944387" Nov 03 22:30:11.392775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.387668 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:11.430503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.427212 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:11.533462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.533427 4625 generic.go:356] "Generic (PLEG): container finished" podID="c7873daa-811c-4d06-8d90-1bc378f4b149" containerID="8621ca927abd21c9de46c72d69d7af394116de433eeec5b86642fc2cad036055" exitCode=0 Nov 03 22:30:11.533608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.533540 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" event={"ID":"c7873daa-811c-4d06-8d90-1bc378f4b149","Type":"ContainerDied","Data":"8621ca927abd21c9de46c72d69d7af394116de433eeec5b86642fc2cad036055"} Nov 03 22:30:11.535396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.535370 4625 generic.go:356] "Generic (PLEG): container finished" podID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerID="0a02dd42a353dfe9049a7bc78d5375781482523763846b9b8b718d79b9c219a3" exitCode=60 Nov 03 22:30:11.535504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.535477 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerDied","Data":"0a02dd42a353dfe9049a7bc78d5375781482523763846b9b8b718d79b9c219a3"} Nov 03 22:30:11.535840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.535827 4625 scope.go:122] "RemoveContainer" containerID="0a02dd42a353dfe9049a7bc78d5375781482523763846b9b8b718d79b9c219a3" Nov 03 22:30:11.553012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.549401 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerStarted","Data":"c942b233b31296f975ae885e30b62e26c537407fb43274399314bf312c567644"} Nov 03 22:30:11.554985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.554959 4625 generic.go:356] "Generic (PLEG): container finished" podID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerID="7a842635519a00a1b551b0f352d49cf853404b7ef5f9a27277d2c4b935299fc9" exitCode=60 Nov 03 22:30:11.555124 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.555097 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerDied","Data":"7a842635519a00a1b551b0f352d49cf853404b7ef5f9a27277d2c4b935299fc9"} Nov 03 22:30:11.555349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.555332 4625 scope.go:122] "RemoveContainer" containerID="7a842635519a00a1b551b0f352d49cf853404b7ef5f9a27277d2c4b935299fc9" Nov 03 22:30:11.565774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.564795 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" event={"ID":"5f24a2a5-4268-4bd4-b505-b5aed592b177","Type":"ContainerStarted","Data":"5afcc407b0b4a9ce67cc129ac139842002ed1451a95aaa40212d22791a0ef78c"} Nov 03 22:30:11.565774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.564858 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:11.565774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.565057 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:11.578780 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.575022 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:30:11.801318 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:11.801253 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" podStartSLOduration=3.4132604730000002 podStartE2EDuration="3.801233697s" totalImagesPullingTime="387.973224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:11.796746057 +0000 UTC m=+1033.226082048" watchObservedRunningTime="2026-11-03 22:30:11.801233697 +0000 UTC m=+1033.230569690" Nov 03 22:30:12.672738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.656022 4625 patch_prober.go:29] interesting pod/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:12.672738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.662825 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" podUID="b963989a-255c-41b8-84a3-f3ffc60141ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.679512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.678133 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerStarted","Data":"afb3de1dd68ae34cd5d249f365b6db06fb3d5ac6fb654573457fe3e9a0173690"} Nov 03 22:30:12.841784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.829618 4625 generic.go:356] "Generic (PLEG): container finished" podID="15f72d4d-599c-49ab-93fb-7b566b020708" containerID="c942b233b31296f975ae885e30b62e26c537407fb43274399314bf312c567644" exitCode=60 Nov 03 22:30:12.841784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.830359 4625 scope.go:122] "RemoveContainer" containerID="c942b233b31296f975ae885e30b62e26c537407fb43274399314bf312c567644" Nov 03 22:30:12.841784 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:12.830846 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g_odh-ai-gateway-infra(15f72d4d-599c-49ab-93fb-7b566b020708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" Nov 03 22:30:12.876178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.875966 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerDied","Data":"c942b233b31296f975ae885e30b62e26c537407fb43274399314bf312c567644"} Nov 03 22:30:12.876178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.876061 4625 scope.go:122] "RemoveContainer" containerID="e3f231a934bfedb0c99db2750b9882696bb8c2f0c076d0ed7d9400b5aab29c19" Nov 03 22:30:12.972400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:12.972344 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerStarted","Data":"8e93a45ca85820ea86b3fe067be2b3b8094e9b845cd65bd511a66177c9410d3d"} Nov 03 22:30:13.206699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.206609 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:13.350503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.350465 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-xks-ca\") pod \"c7873daa-811c-4d06-8d90-1bc378f4b149\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " Nov 03 22:30:13.350707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.350691 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-openshift-ca\") pod \"c7873daa-811c-4d06-8d90-1bc378f4b149\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " Nov 03 22:30:13.350940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.350923 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7873daa-811c-4d06-8d90-1bc378f4b149-kube-api-access-fbf2c\" (UniqueName: \"kubernetes.io/projected/c7873daa-811c-4d06-8d90-1bc378f4b149-kube-api-access-fbf2c\") pod \"c7873daa-811c-4d06-8d90-1bc378f4b149\" (UID: \"c7873daa-811c-4d06-8d90-1bc378f4b149\") " Nov 03 22:30:13.351365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.351111 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-openshift-ca" pod "c7873daa-811c-4d06-8d90-1bc378f4b149" (UID: "c7873daa-811c-4d06-8d90-1bc378f4b149"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:13.351704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.351684 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:13.353248 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.353221 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-xks-ca" pod "c7873daa-811c-4d06-8d90-1bc378f4b149" (UID: "c7873daa-811c-4d06-8d90-1bc378f4b149"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:13.353502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.353466 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7873daa-811c-4d06-8d90-1bc378f4b149-kube-api-access-fbf2c" pod "c7873daa-811c-4d06-8d90-1bc378f4b149" (UID: "c7873daa-811c-4d06-8d90-1bc378f4b149"). InnerVolumeSpecName "kube-api-access-fbf2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:13.453452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.453409 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7873daa-811c-4d06-8d90-1bc378f4b149-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:13.453452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.453444 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbf2c\" (UniqueName: \"kubernetes.io/projected/c7873daa-811c-4d06-8d90-1bc378f4b149-kube-api-access-fbf2c\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:13.989302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.989263 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" Nov 03 22:30:13.989768 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.989346 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc" event={"ID":"c7873daa-811c-4d06-8d90-1bc378f4b149","Type":"ContainerDied","Data":"4974f3918b9dc0ffc41f9356d5255c4ffc997d5272f0e00832770633c58e533c"} Nov 03 22:30:13.989768 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.989421 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4974f3918b9dc0ffc41f9356d5255c4ffc997d5272f0e00832770633c58e533c" Nov 03 22:30:13.992579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.992270 4625 generic.go:356] "Generic (PLEG): container finished" podID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerID="afb3de1dd68ae34cd5d249f365b6db06fb3d5ac6fb654573457fe3e9a0173690" exitCode=60 Nov 03 22:30:13.992579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.992455 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerDied","Data":"afb3de1dd68ae34cd5d249f365b6db06fb3d5ac6fb654573457fe3e9a0173690"} Nov 03 22:30:13.992579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.992539 4625 scope.go:122] "RemoveContainer" containerID="0a02dd42a353dfe9049a7bc78d5375781482523763846b9b8b718d79b9c219a3" Nov 03 22:30:13.994643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.994614 4625 scope.go:122] "RemoveContainer" containerID="afb3de1dd68ae34cd5d249f365b6db06fb3d5ac6fb654573457fe3e9a0173690" Nov 03 22:30:13.995083 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:13.995054 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk_odh-ai-gateway-infra(a7e87482-394e-481d-993b-590fd0a6eb3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" Nov 03 22:30:13.999928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:13.999902 4625 scope.go:122] "RemoveContainer" containerID="c942b233b31296f975ae885e30b62e26c537407fb43274399314bf312c567644" Nov 03 22:30:14.000237 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:14.000207 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g_odh-ai-gateway-infra(15f72d4d-599c-49ab-93fb-7b566b020708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" Nov 03 22:30:14.002577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.002551 4625 generic.go:356] "Generic (PLEG): container finished" podID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerID="8e93a45ca85820ea86b3fe067be2b3b8094e9b845cd65bd511a66177c9410d3d" exitCode=60 Nov 03 22:30:14.003629 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.003607 4625 scope.go:122] "RemoveContainer" containerID="8e93a45ca85820ea86b3fe067be2b3b8094e9b845cd65bd511a66177c9410d3d" Nov 03 22:30:14.003924 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:14.003898 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4_odh-ai-gateway-infra(c8934af8-8db7-4ce4-a899-7abdfef8a34f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" Nov 03 22:30:14.004038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.003948 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerDied","Data":"8e93a45ca85820ea86b3fe067be2b3b8094e9b845cd65bd511a66177c9410d3d"} Nov 03 22:30:14.015520 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.015480 4625 scope.go:122] "RemoveContainer" containerID="7a842635519a00a1b551b0f352d49cf853404b7ef5f9a27277d2c4b935299fc9" Nov 03 22:30:14.164808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.164754 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v"] Nov 03 22:30:14.166767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.166002 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c7873daa-811c-4d06-8d90-1bc378f4b149" containerName="revoke-keys" Nov 03 22:30:14.166767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.166028 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7873daa-811c-4d06-8d90-1bc378f4b149" containerName="revoke-keys" Nov 03 22:30:14.166767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.166255 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7873daa-811c-4d06-8d90-1bc378f4b149" containerName="revoke-keys" Nov 03 22:30:14.171620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.171597 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.196153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.196117 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v"] Nov 03 22:30:14.223914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.223857 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6"] Nov 03 22:30:14.229496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.229468 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.262778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.262646 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6"] Nov 03 22:30:14.267026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.266989 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e504378b-605b-45b9-8573-33b540fafc36-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.267161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.267045 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.267224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.267199 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e504378b-605b-45b9-8573-33b540fafc36-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.267285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.267255 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmd5d\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-kube-api-access-xmd5d\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.268220 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.268191 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.268470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.268253 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.268470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.268283 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.268470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.268409 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.268470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.268474 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.460738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.426635 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.461065 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.461030 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.478755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.467856 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e504378b-605b-45b9-8573-33b540fafc36-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.505753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.486812 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.509126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.506201 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.509821 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.490236 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.509963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.509797 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.509963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.509908 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e504378b-605b-45b9-8573-33b540fafc36-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.511545 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.511485 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e504378b-605b-45b9-8573-33b540fafc36-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.536798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.525938 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.536798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.526025 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmd5d\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-kube-api-access-xmd5d\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.537058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.434296 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.612747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.612357 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.612747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.612457 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldndg\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-kube-api-access-ldndg\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.612747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.612505 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.612747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.612623 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.612747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.612679 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.612747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.612742 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.613237 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.613037 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.613237 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.613076 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.613237 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.613227 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.670740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.618926 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.670740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.619268 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.749747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.718044 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.811504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.811385 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.812501 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.812466 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.812680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.812665 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.816731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.813492 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.816904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.751933 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.817017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.812284 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.817304 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.817285 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.817480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.817460 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldndg\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-kube-api-access-ldndg\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.817599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.817585 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.821963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.821918 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.822194 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.822174 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.843922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.822790 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.844239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.823115 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.844390 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.823382 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.859374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.859077 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.862770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.862734 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmd5d\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-kube-api-access-xmd5d\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.868583 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.868510 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldndg\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-kube-api-access-ldndg\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.875787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.875748 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e504378b-605b-45b9-8573-33b540fafc36-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.876890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.876617 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:14.882994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.878820 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:14.889547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:14.889499 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:15.016398 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.016363 4625 scope.go:122] "RemoveContainer" containerID="afb3de1dd68ae34cd5d249f365b6db06fb3d5ac6fb654573457fe3e9a0173690" Nov 03 22:30:15.016898 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:15.016870 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk_odh-ai-gateway-infra(a7e87482-394e-481d-993b-590fd0a6eb3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" Nov 03 22:30:15.018203 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.018179 4625 scope.go:122] "RemoveContainer" containerID="8e93a45ca85820ea86b3fe067be2b3b8094e9b845cd65bd511a66177c9410d3d" Nov 03 22:30:15.018572 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:15.018550 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4_odh-ai-gateway-infra(c8934af8-8db7-4ce4-a899-7abdfef8a34f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" Nov 03 22:30:15.096693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.096655 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:15.151106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.151064 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:15.516269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.516169 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v"] Nov 03 22:30:15.538281 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:30:15.538242 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode504378b_605b_45b9_8573_33b540fafc36.slice/crio-06deb7756bc926a2d5b5fb80914971d873a933e98bb63c3492c18b0459c64209 WatchSource:0}: Error finding container 06deb7756bc926a2d5b5fb80914971d873a933e98bb63c3492c18b0459c64209: Status 404 returned error can't find the container with id 06deb7756bc926a2d5b5fb80914971d873a933e98bb63c3492c18b0459c64209 Nov 03 22:30:15.542243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.542200 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:15.542341 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.542293 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:15.542413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.542351 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:15.578839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.577001 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6"] Nov 03 22:30:15.587009 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:30:15.586975 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47eef8a9_d72d_470d_9b3a_622ae9bf55e9.slice/crio-0e97cd0bd2416fb319f31326d14414ad0ad07ef17b2fd3a2a83b96efad194d33 WatchSource:0}: Error finding container 0e97cd0bd2416fb319f31326d14414ad0ad07ef17b2fd3a2a83b96efad194d33: Status 404 returned error can't find the container with id 0e97cd0bd2416fb319f31326d14414ad0ad07ef17b2fd3a2a83b96efad194d33 Nov 03 22:30:15.590322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.590281 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:15.590430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.590359 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:15.590430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:15.590389 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:30:16.030582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.029625 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" event={"ID":"47eef8a9-d72d-470d-9b3a-622ae9bf55e9","Type":"ContainerStarted","Data":"2d44c4e822a0567271e169c9273b1c2e8712b9bd1348d2b63774564565473146"} Nov 03 22:30:16.030582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.029700 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" event={"ID":"47eef8a9-d72d-470d-9b3a-622ae9bf55e9","Type":"ContainerStarted","Data":"0e97cd0bd2416fb319f31326d14414ad0ad07ef17b2fd3a2a83b96efad194d33"} Nov 03 22:30:16.033364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.033331 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" event={"ID":"e504378b-605b-45b9-8573-33b540fafc36","Type":"ContainerStarted","Data":"286ba0710df0fb2e36aba63b4df4b4634a8996144bf1d3912497a50bd4c6aa57"} Nov 03 22:30:16.033600 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.033385 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" event={"ID":"e504378b-605b-45b9-8573-33b540fafc36","Type":"ContainerStarted","Data":"06deb7756bc926a2d5b5fb80914971d873a933e98bb63c3492c18b0459c64209"} Nov 03 22:30:16.077396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.077342 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podStartSLOduration=2.077323591 podStartE2EDuration="2.077323591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:16.074445221 +0000 UTC m=+1037.503781214" watchObservedRunningTime="2026-11-03 22:30:16.077323591 +0000 UTC m=+1037.506659583" Nov 03 22:30:16.097342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.097302 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:16.098938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.098911 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.099075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.098961 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:16.120506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.120439 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podStartSLOduration=2.120418393 podStartE2EDuration="2.120418393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:16.118568109 +0000 UTC m=+1037.547904102" watchObservedRunningTime="2026-11-03 22:30:16.120418393 +0000 UTC m=+1037.549754388" Nov 03 22:30:16.153401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.153312 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:16.191740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.187119 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.191740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:16.187193 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:17.098131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:17.097565 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.098131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:17.097638 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:17.175754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:17.162029 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.175754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:17.162091 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:18.223927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.223683 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:18.225081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.224515 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:18.229388 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.227423 4625 generic.go:356] "Generic (PLEG): container finished" podID="b963989a-255c-41b8-84a3-f3ffc60141ac" containerID="163585c2a9aa501781237492aa63e5c3d02118c4517d9dacd820a62d97768de9" exitCode=0 Nov 03 22:30:18.229388 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.228965 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" event={"ID":"b963989a-255c-41b8-84a3-f3ffc60141ac","Type":"ContainerDied","Data":"163585c2a9aa501781237492aa63e5c3d02118c4517d9dacd820a62d97768de9"} Nov 03 22:30:18.321921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.218816 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:18.322108 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.321916 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:18.328037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.328003 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:30:18.527545 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.527455 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-certs\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.527699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.527579 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b963989a-255c-41b8-84a3-f3ffc60141ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b963989a-255c-41b8-84a3-f3ffc60141ac-istiod-ca-cert\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.527793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.527765 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-certs" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:18.528026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528005 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b963989a-255c-41b8-84a3-f3ffc60141ac-istiod-ca-cert" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:18.528158 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528140 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-token\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-token\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.528234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528203 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-socket\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.528294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528279 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-credential-socket\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.528379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528334 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-podinfo\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.528434 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528409 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-data\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.528590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528562 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-credential-socket" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:18.528590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528575 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-socket" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:18.528806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528612 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-kube-api-access-9jsp8\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-kube-api-access-9jsp8\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.528806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528646 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-envoy\") pod \"b963989a-255c-41b8-84a3-f3ffc60141ac\" (UID: \"b963989a-255c-41b8-84a3-f3ffc60141ac\") " Nov 03 22:30:18.528928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.528832 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-data" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:18.529512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.529432 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.529512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.529461 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b963989a-255c-41b8-84a3-f3ffc60141ac-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.529512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.529474 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.529512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.529486 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.529512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.529500 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.530857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.530819 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-token" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:18.531354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.531328 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-podinfo" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:30:18.531420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.531398 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-envoy" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:18.531842 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.531813 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-kube-api-access-9jsp8" pod "b963989a-255c-41b8-84a3-f3ffc60141ac" (UID: "b963989a-255c-41b8-84a3-f3ffc60141ac"). InnerVolumeSpecName "kube-api-access-9jsp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:18.615701 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:18.615644 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:18.630790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.630760 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.630790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.630786 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.630790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.630796 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jsp8\" (UniqueName: \"kubernetes.io/projected/b963989a-255c-41b8-84a3-f3ffc60141ac-kube-api-access-9jsp8\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.631057 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.630805 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b963989a-255c-41b8-84a3-f3ffc60141ac-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:18.907038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:18.907009 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j"] Nov 03 22:30:19.016702 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.016423 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:30:19.100593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.100559 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:19.100876 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.100846 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:19.157793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.155835 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:19.157793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.155893 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:19.294241 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.294203 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" event={"ID":"b963989a-255c-41b8-84a3-f3ffc60141ac","Type":"ContainerDied","Data":"c9cf66d35512f0458d78c720bce07478c0075caa279745771d859f1f855eba8c"} Nov 03 22:30:19.294830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.294808 4625 scope.go:122] "RemoveContainer" containerID="163585c2a9aa501781237492aa63e5c3d02118c4517d9dacd820a62d97768de9" Nov 03 22:30:19.295060 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.294474 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88" Nov 03 22:30:19.804744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:19.801787 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88"] Nov 03 22:30:20.046760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.044260 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1b279244-openshift-default-74dd88dd99-6px88"] Nov 03 22:30:20.120739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.112591 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:20.120739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.112650 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:20.156590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.156453 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:20.156590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.156517 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:20.221177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.207894 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc"] Nov 03 22:30:20.221177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.209153 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b963989a-255c-41b8-84a3-f3ffc60141ac" containerName="istio-proxy" Nov 03 22:30:20.221177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.209175 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b963989a-255c-41b8-84a3-f3ffc60141ac" containerName="istio-proxy" Nov 03 22:30:20.221177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.209408 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b963989a-255c-41b8-84a3-f3ffc60141ac" containerName="istio-proxy" Nov 03 22:30:20.288489 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.278007 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.307237 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.300692 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d05919ef-serving-cert\"" Nov 03 22:30:20.372496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.371943 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-metrics-certs\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.372496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.372041 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828e950a-ac6a-45ed-abb7-b11753927f23-openshift-service-ca\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.372496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.372096 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7p88\" (UniqueName: \"kubernetes.io/projected/828e950a-ac6a-45ed-abb7-b11753927f23-kube-api-access-m7p88\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.372496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.372239 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-maas-api-tls\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.386922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.386820 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc"] Nov 03 22:30:20.715818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.714869 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:20.773196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.772747 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.835926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.833951 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-maas-api-tls\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.835926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.834370 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-metrics-certs\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.835926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.834514 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828e950a-ac6a-45ed-abb7-b11753927f23-openshift-service-ca\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.835926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.834623 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7p88\" (UniqueName: \"kubernetes.io/projected/828e950a-ac6a-45ed-abb7-b11753927f23-kube-api-access-m7p88\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.864179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.864142 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828e950a-ac6a-45ed-abb7-b11753927f23-openshift-service-ca\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:20.900743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.884243 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b963989a-255c-41b8-84a3-f3ffc60141ac" path="/var/lib/kubelet/pods/b963989a-255c-41b8-84a3-f3ffc60141ac/volumes" Nov 03 22:30:20.950889 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:20.946098 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7p88\" (UniqueName: \"kubernetes.io/projected/828e950a-ac6a-45ed-abb7-b11753927f23-kube-api-access-m7p88\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:21.043816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.041126 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-maas-api-tls\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:21.311863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.307558 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-metrics-certs\") pod \"maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:21.332191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.325994 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:21.332191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.326084 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:21.332191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.327361 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:21.349963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.349853 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:21.350139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.350014 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:21.527743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.523909 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6"] Nov 03 22:30:21.527743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.524204 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" podUID="3f7dcea8-98ec-46c4-8a33-84abca5f6c96" containerName="maas-api" containerID="cri-o://c485914ecc6afb976bcacbe25bf76f66b8a0dd36626999537fb01211c5d8b08c" gracePeriod=30 Nov 03 22:30:21.859087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.831054 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.887080 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:21.887048 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:30:22.196329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.196216 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.237749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.205227 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:30:22.328744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.230360 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:22.384860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.384809 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:22.425995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.419307 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc"] Nov 03 22:30:22.425995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.335190 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.457825 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:22.268017 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:22.504741 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:22.282296 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:22.578744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.575887 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:22.578744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.575965 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:22.695740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:22.648060 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.107259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.107209 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.336015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.335984 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:23.336660 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.336618 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:23.338705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.311103 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:23.368601 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.367498 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:23.368601 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.367553 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:23.413544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.411291 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:30:23.477754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.470624 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-7c5g6" podUID="6f8dc406-d4b0-407a-bd99-069aa88016cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.477754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.472607 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.643107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.632389 4625 generic.go:356] "Generic (PLEG): container finished" podID="3f7dcea8-98ec-46c4-8a33-84abca5f6c96" containerID="c485914ecc6afb976bcacbe25bf76f66b8a0dd36626999537fb01211c5d8b08c" exitCode=0 Nov 03 22:30:23.643107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.635693 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" event={"ID":"3f7dcea8-98ec-46c4-8a33-84abca5f6c96","Type":"ContainerDied","Data":"c485914ecc6afb976bcacbe25bf76f66b8a0dd36626999537fb01211c5d8b08c"} Nov 03 22:30:23.714620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.714580 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51bfa2e1-v62cc"] Nov 03 22:30:23.715016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.714995 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:30:23.717337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.717310 4625 scope.go:122] "RemoveContainer" containerID="8e93a45ca85820ea86b3fe067be2b3b8094e9b845cd65bd511a66177c9410d3d" Nov 03 22:30:23.732789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.718154 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:30:23.732789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.718196 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:30:23.765751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.752274 4625 scope.go:122] "RemoveContainer" containerID="afb3de1dd68ae34cd5d249f365b6db06fb3d5ac6fb654573457fe3e9a0173690" Nov 03 22:30:23.917259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:23.908193 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" podUID="7fcdf9ce-4504-4143-8dea-84a702e155e9" containerName="payload-pre-processing" containerID="cri-o://f938bfd5b0e8fb2154b3cfa097ae785167984be3b267c54f3e1888c8ec35d604" gracePeriod=25 Nov 03 22:30:24.052754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.050703 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb"] Nov 03 22:30:24.169078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.168989 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:24.169317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.169288 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:24.345278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.345245 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:24.346597 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.346572 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:30:24.347061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.347033 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.476836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.476746 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-metrics-certs\") pod \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " Nov 03 22:30:24.477104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.477082 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-kube-api-access-bklcl\" (UniqueName: \"kubernetes.io/projected/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-kube-api-access-bklcl\") pod \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " Nov 03 22:30:24.477301 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.477284 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-openshift-service-ca\") pod \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " Nov 03 22:30:24.477583 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.477533 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-maas-api-tls\") pod \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\" (UID: \"3f7dcea8-98ec-46c4-8a33-84abca5f6c96\") " Nov 03 22:30:24.486535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.482235 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:24.501525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.501495 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ec3023-t25zb"] Nov 03 22:30:24.507529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.506974 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-openshift-service-ca" pod "3f7dcea8-98ec-46c4-8a33-84abca5f6c96" (UID: "3f7dcea8-98ec-46c4-8a33-84abca5f6c96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:24.510431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.510388 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-kube-api-access-bklcl" pod "3f7dcea8-98ec-46c4-8a33-84abca5f6c96" (UID: "3f7dcea8-98ec-46c4-8a33-84abca5f6c96"). InnerVolumeSpecName "kube-api-access-bklcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:24.510817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.510795 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-metrics-certs" pod "3f7dcea8-98ec-46c4-8a33-84abca5f6c96" (UID: "3f7dcea8-98ec-46c4-8a33-84abca5f6c96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:24.533064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.531676 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9"] Nov 03 22:30:24.533064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.531987 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" podUID="60521624-9dd2-42e5-b0d0-9fd65d597179" containerName="istio-proxy" containerID="cri-o://630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868" gracePeriod=30 Nov 03 22:30:24.533064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.532282 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-maas-api-tls" pod "3f7dcea8-98ec-46c4-8a33-84abca5f6c96" (UID: "3f7dcea8-98ec-46c4-8a33-84abca5f6c96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:24.533064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.532658 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:30:24.595177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.593973 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc"] Nov 03 22:30:24.601905 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.601877 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:24.602208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.602191 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:24.602393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.602377 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklcl\" (UniqueName: \"kubernetes.io/projected/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-kube-api-access-bklcl\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:24.602549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.602532 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dcea8-98ec-46c4-8a33-84abca5f6c96-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:24.826152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.826113 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7873daa-811c-4d06-8d90-1bc378f4b149" path="/var/lib/kubelet/pods/c7873daa-811c-4d06-8d90-1bc378f4b149/volumes" Nov 03 22:30:24.828199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.828176 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d" path="/var/lib/kubelet/pods/ceef80a8-ad4e-4df9-afcd-7f8f4d46fe8d/volumes" Nov 03 22:30:24.843891 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.843848 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" event={"ID":"828e950a-ac6a-45ed-abb7-b11753927f23","Type":"ContainerStarted","Data":"8c4343e1cf5ae07f2a27a5f81d036fd93eebf33ed3926276f0ee58909d99b8a1"} Nov 03 22:30:24.846558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.846525 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" event={"ID":"3f7dcea8-98ec-46c4-8a33-84abca5f6c96","Type":"ContainerDied","Data":"8427ee9e7b7cfa3cf5d7d7175fb275db338b10dc3413c1edc34911da0655c92a"} Nov 03 22:30:24.850248 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.850223 4625 scope.go:122] "RemoveContainer" containerID="c485914ecc6afb976bcacbe25bf76f66b8a0dd36626999537fb01211c5d8b08c" Nov 03 22:30:24.850562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.846882 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" Nov 03 22:30:24.855291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.855267 4625 generic.go:356] "Generic (PLEG): container finished" podID="7fcdf9ce-4504-4143-8dea-84a702e155e9" containerID="f938bfd5b0e8fb2154b3cfa097ae785167984be3b267c54f3e1888c8ec35d604" exitCode=0 Nov 03 22:30:24.856969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.855432 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" event={"ID":"7fcdf9ce-4504-4143-8dea-84a702e155e9","Type":"ContainerDied","Data":"f938bfd5b0e8fb2154b3cfa097ae785167984be3b267c54f3e1888c8ec35d604"} Nov 03 22:30:24.894847 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:24.894805 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcdf9ce_4504_4143_8dea_84a702e155e9.slice/crio-f938bfd5b0e8fb2154b3cfa097ae785167984be3b267c54f3e1888c8ec35d604.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcdf9ce_4504_4143_8dea_84a702e155e9.slice/crio-conmon-f938bfd5b0e8fb2154b3cfa097ae785167984be3b267c54f3e1888c8ec35d604.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:24.954501 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.954472 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6"] Nov 03 22:30:24.961778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:24.961751 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6"] Nov 03 22:30:25.054523 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.054473 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:30:25.097785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.097434 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 22:30:25.097785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.097495 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 22:30:25.119379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.119350 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcdf9ce-4504-4143-8dea-84a702e155e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdf9ce-4504-4143-8dea-84a702e155e9-plugins-config-volume\") pod \"7fcdf9ce-4504-4143-8dea-84a702e155e9\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " Nov 03 22:30:25.119882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.119851 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcdf9ce-4504-4143-8dea-84a702e155e9-kube-api-access-g7b5g\" (UniqueName: \"kubernetes.io/projected/7fcdf9ce-4504-4143-8dea-84a702e155e9-kube-api-access-g7b5g\") pod \"7fcdf9ce-4504-4143-8dea-84a702e155e9\" (UID: \"7fcdf9ce-4504-4143-8dea-84a702e155e9\") " Nov 03 22:30:25.123051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.123008 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcdf9ce-4504-4143-8dea-84a702e155e9-plugins-config-volume" pod "7fcdf9ce-4504-4143-8dea-84a702e155e9" (UID: "7fcdf9ce-4504-4143-8dea-84a702e155e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:25.124139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.124100 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcdf9ce-4504-4143-8dea-84a702e155e9-kube-api-access-g7b5g" pod "7fcdf9ce-4504-4143-8dea-84a702e155e9" (UID: "7fcdf9ce-4504-4143-8dea-84a702e155e9"). InnerVolumeSpecName "kube-api-access-g7b5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:25.152226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.152199 4625 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:30:25.152419 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.152392 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 22:30:25.223100 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.223063 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdf9ce-4504-4143-8dea-84a702e155e9-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:25.223100 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.223102 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7b5g\" (UniqueName: \"kubernetes.io/projected/7fcdf9ce-4504-4143-8dea-84a702e155e9-kube-api-access-g7b5g\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:25.727805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.727363 4625 scope.go:122] "RemoveContainer" containerID="c942b233b31296f975ae885e30b62e26c537407fb43274399314bf312c567644" Nov 03 22:30:25.905622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.905431 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerStarted","Data":"0a1aee99136fed4526c4e7893c1943e4b3b2941e8e5704a3644de13fce4057fa"} Nov 03 22:30:25.922396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.922360 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" event={"ID":"7fcdf9ce-4504-4143-8dea-84a702e155e9","Type":"ContainerDied","Data":"7db8c0a81e0f1a132b6ff6fb17ff3ea3d71526f72ba1424b2b8d801278b2ae46"} Nov 03 22:30:25.922665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.922642 4625 scope.go:122] "RemoveContainer" containerID="f938bfd5b0e8fb2154b3cfa097ae785167984be3b267c54f3e1888c8ec35d604" Nov 03 22:30:25.922935 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.922918 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j" Nov 03 22:30:25.931280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.930411 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerStarted","Data":"9e9b31c3abc005e7d19504092397805a8df88819da7925356992eb30dcb12b70"} Nov 03 22:30:25.940412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.940383 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" event={"ID":"828e950a-ac6a-45ed-abb7-b11753927f23","Type":"ContainerStarted","Data":"f8c7ac3c6282b48348a886b829e49ffc27b3fb3c4f008aa16dd2de50c1b592aa"} Nov 03 22:30:25.940825 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:25.940767 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:26.106134 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.106069 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" podStartSLOduration=6.11276451 podStartE2EDuration="7.106049466s" totalImagesPullingTime="993.284956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:26.067567365 +0000 UTC m=+1047.496903360" watchObservedRunningTime="2026-11-03 22:30:26.106049466 +0000 UTC m=+1047.535385460" Nov 03 22:30:26.110557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.110529 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j"] Nov 03 22:30:26.123416 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.123379 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51bfa2e1-559ff7c87f-ckt8j"] Nov 03 22:30:26.492739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.468116 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-51bfa2e1-8698958694-88fk6" podUID="3f7dcea8-98ec-46c4-8a33-84abca5f6c96" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.103:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:26.571632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.571606 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:26.573242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.573167 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:26.574878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.574834 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:30:26.771878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.771766 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7dcea8-98ec-46c4-8a33-84abca5f6c96" path="/var/lib/kubelet/pods/3f7dcea8-98ec-46c4-8a33-84abca5f6c96/volumes" Nov 03 22:30:26.773162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.773120 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fcdf9ce-4504-4143-8dea-84a702e155e9" path="/var/lib/kubelet/pods/7fcdf9ce-4504-4143-8dea-84a702e155e9/volumes" Nov 03 22:30:26.841572 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.841537 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:26.843052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.843024 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:26.843577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.843556 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:30:26.956587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.956549 4625 generic.go:356] "Generic (PLEG): container finished" podID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerID="9e9b31c3abc005e7d19504092397805a8df88819da7925356992eb30dcb12b70" exitCode=60 Nov 03 22:30:26.956831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.956643 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerDied","Data":"9e9b31c3abc005e7d19504092397805a8df88819da7925356992eb30dcb12b70"} Nov 03 22:30:26.956831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.956709 4625 scope.go:122] "RemoveContainer" containerID="afb3de1dd68ae34cd5d249f365b6db06fb3d5ac6fb654573457fe3e9a0173690" Nov 03 22:30:26.957500 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.957207 4625 scope.go:122] "RemoveContainer" containerID="9e9b31c3abc005e7d19504092397805a8df88819da7925356992eb30dcb12b70" Nov 03 22:30:26.957684 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:26.957659 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk_odh-ai-gateway-infra(a7e87482-394e-481d-993b-590fd0a6eb3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" Nov 03 22:30:26.961509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.961482 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerStarted","Data":"48b99308e3f0048dd7f34e4109d383cf0570c23079bce3543d8f33a9ac1cf6c2"} Nov 03 22:30:26.965225 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.965196 4625 generic.go:356] "Generic (PLEG): container finished" podID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerID="0a1aee99136fed4526c4e7893c1943e4b3b2941e8e5704a3644de13fce4057fa" exitCode=60 Nov 03 22:30:26.965347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.965284 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerDied","Data":"0a1aee99136fed4526c4e7893c1943e4b3b2941e8e5704a3644de13fce4057fa"} Nov 03 22:30:26.966093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.965925 4625 scope.go:122] "RemoveContainer" containerID="0a1aee99136fed4526c4e7893c1943e4b3b2941e8e5704a3644de13fce4057fa" Nov 03 22:30:26.966217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.966112 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4"] Nov 03 22:30:26.966364 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:26.966341 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4_odh-ai-gateway-infra(c8934af8-8db7-4ce4-a899-7abdfef8a34f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" Nov 03 22:30:26.974624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:26.974537 4625 scope.go:122] "RemoveContainer" containerID="8e93a45ca85820ea86b3fe067be2b3b8094e9b845cd65bd511a66177c9410d3d" Nov 03 22:30:27.079502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:27.079393 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk"] Nov 03 22:30:27.980287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:27.980248 4625 generic.go:356] "Generic (PLEG): container finished" podID="15f72d4d-599c-49ab-93fb-7b566b020708" containerID="48b99308e3f0048dd7f34e4109d383cf0570c23079bce3543d8f33a9ac1cf6c2" exitCode=60 Nov 03 22:30:27.980287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:27.980284 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerDied","Data":"48b99308e3f0048dd7f34e4109d383cf0570c23079bce3543d8f33a9ac1cf6c2"} Nov 03 22:30:27.980828 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:27.980346 4625 scope.go:122] "RemoveContainer" containerID="c942b233b31296f975ae885e30b62e26c537407fb43274399314bf312c567644" Nov 03 22:30:27.980828 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:27.980814 4625 scope.go:122] "RemoveContainer" containerID="48b99308e3f0048dd7f34e4109d383cf0570c23079bce3543d8f33a9ac1cf6c2" Nov 03 22:30:27.981279 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:27.981251 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g_odh-ai-gateway-infra(15f72d4d-599c-49ab-93fb-7b566b020708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" Nov 03 22:30:28.210377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.210092 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:28.252989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.252955 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g"] Nov 03 22:30:28.270274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.270245 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:28.314175 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.314138 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8934af8-8db7-4ce4-a899-7abdfef8a34f-kube-api-access-l2kzn\" (UniqueName: \"kubernetes.io/projected/c8934af8-8db7-4ce4-a899-7abdfef8a34f-kube-api-access-l2kzn\") pod \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " Nov 03 22:30:28.314375 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.314283 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e87482-394e-481d-993b-590fd0a6eb3e-kube-api-access-98m5p\" (UniqueName: \"kubernetes.io/projected/a7e87482-394e-481d-993b-590fd0a6eb3e-kube-api-access-98m5p\") pod \"a7e87482-394e-481d-993b-590fd0a6eb3e\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " Nov 03 22:30:28.314471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.314451 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e87482-394e-481d-993b-590fd0a6eb3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e87482-394e-481d-993b-590fd0a6eb3e-maas-api-service-ca\") pod \"a7e87482-394e-481d-993b-590fd0a6eb3e\" (UID: \"a7e87482-394e-481d-993b-590fd0a6eb3e\") " Nov 03 22:30:28.315443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.315035 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8934af8-8db7-4ce4-a899-7abdfef8a34f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8934af8-8db7-4ce4-a899-7abdfef8a34f-maas-api-service-ca\") pod \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\" (UID: \"c8934af8-8db7-4ce4-a899-7abdfef8a34f\") " Nov 03 22:30:28.315443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.315408 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8934af8-8db7-4ce4-a899-7abdfef8a34f-maas-api-service-ca" pod "c8934af8-8db7-4ce4-a899-7abdfef8a34f" (UID: "c8934af8-8db7-4ce4-a899-7abdfef8a34f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:28.316031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.316007 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e87482-394e-481d-993b-590fd0a6eb3e-maas-api-service-ca" pod "a7e87482-394e-481d-993b-590fd0a6eb3e" (UID: "a7e87482-394e-481d-993b-590fd0a6eb3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:28.318448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.317646 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e87482-394e-481d-993b-590fd0a6eb3e-maas-api-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:28.318448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.317667 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8934af8-8db7-4ce4-a899-7abdfef8a34f-maas-api-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:28.319765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.319697 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e87482-394e-481d-993b-590fd0a6eb3e-kube-api-access-98m5p" pod "a7e87482-394e-481d-993b-590fd0a6eb3e" (UID: "a7e87482-394e-481d-993b-590fd0a6eb3e"). InnerVolumeSpecName "kube-api-access-98m5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:28.328955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.328927 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8934af8-8db7-4ce4-a899-7abdfef8a34f-kube-api-access-l2kzn" pod "c8934af8-8db7-4ce4-a899-7abdfef8a34f" (UID: "c8934af8-8db7-4ce4-a899-7abdfef8a34f"). InnerVolumeSpecName "kube-api-access-l2kzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:28.419496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.419459 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2kzn\" (UniqueName: \"kubernetes.io/projected/c8934af8-8db7-4ce4-a899-7abdfef8a34f-kube-api-access-l2kzn\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:28.419496 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:28.419490 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98m5p\" (UniqueName: \"kubernetes.io/projected/a7e87482-394e-481d-993b-590fd0a6eb3e-kube-api-access-98m5p\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:28.633899 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:28.633856 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:29.069786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.063345 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" event={"ID":"a7e87482-394e-481d-993b-590fd0a6eb3e","Type":"ContainerDied","Data":"e4e60dea1a3547f96bce561ade9256f625b48f8ddcf52e088f98785f2a6a2d74"} Nov 03 22:30:29.069786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.063427 4625 scope.go:122] "RemoveContainer" containerID="9e9b31c3abc005e7d19504092397805a8df88819da7925356992eb30dcb12b70" Nov 03 22:30:29.069786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.063537 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk" Nov 03 22:30:29.116843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.116807 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" event={"ID":"c8934af8-8db7-4ce4-a899-7abdfef8a34f","Type":"ContainerDied","Data":"0c263e4d44400176f0a18b1f76a19f72747506061b15c214acdc3c94172b0c7f"} Nov 03 22:30:29.117098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.117081 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4" Nov 03 22:30:29.158035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.157909 4625 scope.go:122] "RemoveContainer" containerID="0a1aee99136fed4526c4e7893c1943e4b3b2941e8e5704a3644de13fce4057fa" Nov 03 22:30:29.173460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.173146 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk"] Nov 03 22:30:29.185494 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.185466 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895750-xkwbk"] Nov 03 22:30:29.494420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.494382 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4"] Nov 03 22:30:29.501095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.501066 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895750-nrzn4"] Nov 03 22:30:29.604848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.603430 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:29.695919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.695843 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f72d4d-599c-49ab-93fb-7b566b020708-kube-api-access-fzs7v\" (UniqueName: \"kubernetes.io/projected/15f72d4d-599c-49ab-93fb-7b566b020708-kube-api-access-fzs7v\") pod \"15f72d4d-599c-49ab-93fb-7b566b020708\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " Nov 03 22:30:29.696367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.696340 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f72d4d-599c-49ab-93fb-7b566b020708-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f72d4d-599c-49ab-93fb-7b566b020708-maas-api-service-ca\") pod \"15f72d4d-599c-49ab-93fb-7b566b020708\" (UID: \"15f72d4d-599c-49ab-93fb-7b566b020708\") " Nov 03 22:30:29.696876 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.696844 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f72d4d-599c-49ab-93fb-7b566b020708-maas-api-service-ca" pod "15f72d4d-599c-49ab-93fb-7b566b020708" (UID: "15f72d4d-599c-49ab-93fb-7b566b020708"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:29.697072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.697055 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f72d4d-599c-49ab-93fb-7b566b020708-maas-api-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:29.699757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.698198 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f72d4d-599c-49ab-93fb-7b566b020708-kube-api-access-fzs7v" pod "15f72d4d-599c-49ab-93fb-7b566b020708" (UID: "15f72d4d-599c-49ab-93fb-7b566b020708"). InnerVolumeSpecName "kube-api-access-fzs7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:29.799348 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.799309 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzs7v\" (UniqueName: \"kubernetes.io/projected/15f72d4d-599c-49ab-93fb-7b566b020708-kube-api-access-fzs7v\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:29.832221 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.832196 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:30:29.899961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.899923 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-certs\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900048 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-socket\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900087 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-data\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900162 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-credential-socket\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900209 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-kube-api-access-wjzmj\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-kube-api-access-wjzmj\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900237 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-envoy\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900235 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-certs" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:29.900492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900284 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60521624-9dd2-42e5-b0d0-9fd65d597179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60521624-9dd2-42e5-b0d0-9fd65d597179-istiod-ca-cert\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900333 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-podinfo\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900376 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-socket" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:29.900492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900401 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-data" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:29.900492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900480 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-credential-socket" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:29.900811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900523 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-token\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-token\") pod \"60521624-9dd2-42e5-b0d0-9fd65d597179\" (UID: \"60521624-9dd2-42e5-b0d0-9fd65d597179\") " Nov 03 22:30:29.900879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.900822 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60521624-9dd2-42e5-b0d0-9fd65d597179-istiod-ca-cert" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:29.901383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.901359 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:29.901383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.901384 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:29.901580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.901397 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:29.901580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.901410 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:29.901580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.901422 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60521624-9dd2-42e5-b0d0-9fd65d597179-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:29.903170 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.903139 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-podinfo" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:30:29.903272 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.903206 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-envoy" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:29.903429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.903409 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-token" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:29.903553 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:29.903537 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-kube-api-access-wjzmj" pod "60521624-9dd2-42e5-b0d0-9fd65d597179" (UID: "60521624-9dd2-42e5-b0d0-9fd65d597179"). InnerVolumeSpecName "kube-api-access-wjzmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:30.002693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.002602 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzmj\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-kube-api-access-wjzmj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:30.002693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.002638 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:30.002693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.002648 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:30.002693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.002656 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60521624-9dd2-42e5-b0d0-9fd65d597179-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:30.125982 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.125946 4625 generic.go:356] "Generic (PLEG): container finished" podID="60521624-9dd2-42e5-b0d0-9fd65d597179" containerID="630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868" exitCode=0 Nov 03 22:30:30.126373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.125998 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" Nov 03 22:30:30.126373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.126026 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" event={"ID":"60521624-9dd2-42e5-b0d0-9fd65d597179","Type":"ContainerDied","Data":"630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868"} Nov 03 22:30:30.126373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.126082 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9" event={"ID":"60521624-9dd2-42e5-b0d0-9fd65d597179","Type":"ContainerDied","Data":"6022561e80937acdaadc1b11b056d481132f4eee5d61033987915ccba1a6238c"} Nov 03 22:30:30.126373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.126109 4625 scope.go:122] "RemoveContainer" containerID="630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868" Nov 03 22:30:30.129028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.129002 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" Nov 03 22:30:30.129155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.129001 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g" event={"ID":"15f72d4d-599c-49ab-93fb-7b566b020708","Type":"ContainerDied","Data":"46613e7aec06a89b432eed6f24746baa2e3d259449da69a3f949cc3400a071c8"} Nov 03 22:30:30.140317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.140288 4625 scope.go:122] "RemoveContainer" containerID="630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868" Nov 03 22:30:30.140669 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:30.140641 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868\": container with ID starting with 630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868 not found: ID does not exist" containerID="630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868" Nov 03 22:30:30.140822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.140680 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868"} err="failed to get container status \"630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868\": rpc error: code = NotFound desc = could not find container \"630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868\": container with ID starting with 630ebea1c77c0ff652a2dd6542eb9814a24158a718f076242c5e99b15bb93868 not found: ID does not exist" Nov 03 22:30:30.140822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.140705 4625 scope.go:122] "RemoveContainer" containerID="48b99308e3f0048dd7f34e4109d383cf0570c23079bce3543d8f33a9ac1cf6c2" Nov 03 22:30:30.181447 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.181417 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9"] Nov 03 22:30:30.191756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.191690 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-51bfa2e1-openshift-default-6dcb64d7dc-7tnb9"] Nov 03 22:30:30.222882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.222844 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g"] Nov 03 22:30:30.227384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.227359 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895750-9546g"] Nov 03 22:30:30.626983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.626948 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9"] Nov 03 22:30:30.627817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627793 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:30:30.627817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627814 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627827 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627832 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627844 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627849 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627862 4625 cpu_manager.go:414] "containerMap: removing container" podUID="60521624-9dd2-42e5-b0d0-9fd65d597179" containerName="istio-proxy" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627866 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60521624-9dd2-42e5-b0d0-9fd65d597179" containerName="istio-proxy" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627875 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7fcdf9ce-4504-4143-8dea-84a702e155e9" containerName="payload-pre-processing" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627879 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fcdf9ce-4504-4143-8dea-84a702e155e9" containerName="payload-pre-processing" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627889 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627893 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627917 4625 cpu_manager.go:414] "containerMap: removing container" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627922 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627934 4625 cpu_manager.go:414] "containerMap: removing container" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627939 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627949 4625 cpu_manager.go:414] "containerMap: removing container" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627953 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627965 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3f7dcea8-98ec-46c4-8a33-84abca5f6c96" containerName="maas-api" Nov 03 22:30:30.627983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.627969 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7dcea8-98ec-46c4-8a33-84abca5f6c96" containerName="maas-api" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628086 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628094 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628100 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628108 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7dcea8-98ec-46c4-8a33-84abca5f6c96" containerName="maas-api" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628114 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628120 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628128 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="60521624-9dd2-42e5-b0d0-9fd65d597179" containerName="istio-proxy" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628135 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fcdf9ce-4504-4143-8dea-84a702e155e9" containerName="payload-pre-processing" Nov 03 22:30:30.628505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.628141 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:30:30.631468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.631450 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.677336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.677110 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9"] Nov 03 22:30:30.713403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.713370 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4ba330-3769-4b16-956f-817c57000c81-openshift-service-ca\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.713579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.713520 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-metrics-certs\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.713632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.713581 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-maas-api-tls\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.713806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.713789 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7262\" (UniqueName: \"kubernetes.io/projected/1b4ba330-3769-4b16-956f-817c57000c81-kube-api-access-x7262\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.730000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.729972 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cbe6e297-serving-cert\"" Nov 03 22:30:30.733366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.733335 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" path="/var/lib/kubelet/pods/15f72d4d-599c-49ab-93fb-7b566b020708/volumes" Nov 03 22:30:30.734032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.734010 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60521624-9dd2-42e5-b0d0-9fd65d597179" path="/var/lib/kubelet/pods/60521624-9dd2-42e5-b0d0-9fd65d597179/volumes" Nov 03 22:30:30.734522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.734502 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" path="/var/lib/kubelet/pods/a7e87482-394e-481d-993b-590fd0a6eb3e/volumes" Nov 03 22:30:30.735383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.735361 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" path="/var/lib/kubelet/pods/c8934af8-8db7-4ce4-a899-7abdfef8a34f/volumes" Nov 03 22:30:30.864741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.848909 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-maas-api-tls\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.864741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.861680 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7262\" (UniqueName: \"kubernetes.io/projected/1b4ba330-3769-4b16-956f-817c57000c81-kube-api-access-x7262\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.864741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.862328 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4ba330-3769-4b16-956f-817c57000c81-openshift-service-ca\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.864741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.862515 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-metrics-certs\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.864741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.863581 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4ba330-3769-4b16-956f-817c57000c81-openshift-service-ca\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.882693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.880122 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-maas-api-tls\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.896770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.892262 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-metrics-certs\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.901528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.898283 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7262\" (UniqueName: \"kubernetes.io/projected/1b4ba330-3769-4b16-956f-817c57000c81-kube-api-access-x7262\") pod \"maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:30.973798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:30.965544 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:31.330645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:31.330612 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9"] Nov 03 22:30:31.331519 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:30:31.331491 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4ba330_3769_4b16_956f_817c57000c81.slice/crio-0a85463a406b28a1efdb1e9b07d10280951d3567fb46661099903c6b1bdf70e1 WatchSource:0}: Error finding container 0a85463a406b28a1efdb1e9b07d10280951d3567fb46661099903c6b1bdf70e1: Status 404 returned error can't find the container with id 0a85463a406b28a1efdb1e9b07d10280951d3567fb46661099903c6b1bdf70e1 Nov 03 22:30:32.040739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:32.018735 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:32.251887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:32.242841 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" event={"ID":"1b4ba330-3769-4b16-956f-817c57000c81","Type":"ContainerStarted","Data":"c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e"} Nov 03 22:30:32.251887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:32.242926 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" event={"ID":"1b4ba330-3769-4b16-956f-817c57000c81","Type":"ContainerStarted","Data":"0a85463a406b28a1efdb1e9b07d10280951d3567fb46661099903c6b1bdf70e1"} Nov 03 22:30:32.251887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:32.244349 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:32.282940 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:32.282901 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:30:32.338063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:32.337987 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" podStartSLOduration=1.988072456 podStartE2EDuration="2.33797185s" totalImagesPullingTime="349.899394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:32.333111386 +0000 UTC m=+1053.762447380" watchObservedRunningTime="2026-11-03 22:30:32.33797185 +0000 UTC m=+1053.767307845" Nov 03 22:30:39.160782 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:39.158250 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:39.160782 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:39.158341 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:39.401957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:39.401928 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:30:41.082833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:41.082784 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms"] Nov 03 22:30:41.099748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:41.093041 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f651cd-hscms"] Nov 03 22:30:42.064742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:42.064463 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg"] Nov 03 22:30:42.120741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:42.120045 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4502c7d3-n6nfg"] Nov 03 22:30:42.392105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:42.392062 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.539907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:42.539859 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.746901 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:42.746801 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:30:42.748366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:42.748008 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.128741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.123155 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1004577a-94a3-42e3-a224-c3114fb07b33" path="/var/lib/kubelet/pods/1004577a-94a3-42e3-a224-c3114fb07b33/volumes" Nov 03 22:30:43.154741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.145163 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c599fd8d-3bfc-4382-862e-7eab9399e89b" path="/var/lib/kubelet/pods/c599fd8d-3bfc-4382-862e-7eab9399e89b/volumes" Nov 03 22:30:43.570812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.556734 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:30:43.590979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.563215 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:43.590979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.573227 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:30:43.590979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.588120 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:30:43.773961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.773911 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.789980 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.789942 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:30:43.868751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.859833 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:30:43.878784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:43.873495 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:30:44.326945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.323492 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:30:44.595803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.590053 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:44.615205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.615165 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:30:44.693308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.693272 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v828p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:44.713886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.713843 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:44.862904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.862325 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:30:44.926168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.925865 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:30:44.930366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:44.930310 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:30:45.956742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:45.873263 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:30:46.085336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:46.085307 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:30:46.351857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:46.349812 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:49.293287 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:49.293245 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:53.815773 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:53.815711 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a8793d_65b1_4529_acfd_c3ae76f32ba0.slice/crio-conmon-57e0363b191ff76d8b3acb85141c95eb55274b3b3fdd7c4d0c1fc13540f5b2ba.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:55.030598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:55.030561 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4"] Nov 03 22:30:55.041233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:55.041201 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37c0f4-q6bb4"] Nov 03 22:30:56.728889 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:56.728858 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e34492c-6b5f-441b-b240-99ac9d5b58e1" path="/var/lib/kubelet/pods/0e34492c-6b5f-441b-b240-99ac9d5b58e1/volumes" Nov 03 22:30:58.116965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.116933 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc"] Nov 03 22:30:58.118027 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.117994 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" podUID="828e950a-ac6a-45ed-abb7-b11753927f23" containerName="maas-api" containerID="cri-o://f8c7ac3c6282b48348a886b829e49ffc27b3fb3c4f008aa16dd2de50c1b592aa" gracePeriod=30 Nov 03 22:30:58.248048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.248005 4625 generic.go:356] "Generic (PLEG): container finished" podID="828e950a-ac6a-45ed-abb7-b11753927f23" containerID="f8c7ac3c6282b48348a886b829e49ffc27b3fb3c4f008aa16dd2de50c1b592aa" exitCode=0 Nov 03 22:30:58.248369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.248345 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" event={"ID":"828e950a-ac6a-45ed-abb7-b11753927f23","Type":"ContainerDied","Data":"f8c7ac3c6282b48348a886b829e49ffc27b3fb3c4f008aa16dd2de50c1b592aa"} Nov 03 22:30:58.643750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.642243 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:58.895061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.895016 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-metrics-certs\") pod \"828e950a-ac6a-45ed-abb7-b11753927f23\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " Nov 03 22:30:58.896684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.896655 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-maas-api-tls\") pod \"828e950a-ac6a-45ed-abb7-b11753927f23\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " Nov 03 22:30:58.897097 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.897075 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828e950a-ac6a-45ed-abb7-b11753927f23-kube-api-access-m7p88\" (UniqueName: \"kubernetes.io/projected/828e950a-ac6a-45ed-abb7-b11753927f23-kube-api-access-m7p88\") pod \"828e950a-ac6a-45ed-abb7-b11753927f23\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " Nov 03 22:30:58.897806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.897784 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828e950a-ac6a-45ed-abb7-b11753927f23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828e950a-ac6a-45ed-abb7-b11753927f23-openshift-service-ca\") pod \"828e950a-ac6a-45ed-abb7-b11753927f23\" (UID: \"828e950a-ac6a-45ed-abb7-b11753927f23\") " Nov 03 22:30:58.938287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.938246 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828e950a-ac6a-45ed-abb7-b11753927f23-openshift-service-ca" pod "828e950a-ac6a-45ed-abb7-b11753927f23" (UID: "828e950a-ac6a-45ed-abb7-b11753927f23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:58.938830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.938798 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828e950a-ac6a-45ed-abb7-b11753927f23-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:58.985307 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:58.985270 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828e950a-ac6a-45ed-abb7-b11753927f23-kube-api-access-m7p88" pod "828e950a-ac6a-45ed-abb7-b11753927f23" (UID: "828e950a-ac6a-45ed-abb7-b11753927f23"). InnerVolumeSpecName "kube-api-access-m7p88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:59.050742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.047227 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7p88\" (UniqueName: \"kubernetes.io/projected/828e950a-ac6a-45ed-abb7-b11753927f23-kube-api-access-m7p88\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:59.050742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.047406 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-metrics-certs" pod "828e950a-ac6a-45ed-abb7-b11753927f23" (UID: "828e950a-ac6a-45ed-abb7-b11753927f23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:59.050742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.047478 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-maas-api-tls" pod "828e950a-ac6a-45ed-abb7-b11753927f23" (UID: "828e950a-ac6a-45ed-abb7-b11753927f23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:59.149543 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.149505 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:59.149543 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.149540 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828e950a-ac6a-45ed-abb7-b11753927f23-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:30:59.365885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.365777 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" event={"ID":"828e950a-ac6a-45ed-abb7-b11753927f23","Type":"ContainerDied","Data":"8c4343e1cf5ae07f2a27a5f81d036fd93eebf33ed3926276f0ee58909d99b8a1"} Nov 03 22:30:59.372740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.368623 4625 scope.go:122] "RemoveContainer" containerID="f8c7ac3c6282b48348a886b829e49ffc27b3fb3c4f008aa16dd2de50c1b592aa" Nov 03 22:30:59.377580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.373204 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc" Nov 03 22:30:59.506993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.506878 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc"] Nov 03 22:30:59.516943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:30:59.516909 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d05919ef-c64c797b6-jj2bc"] Nov 03 22:30:59.572247 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:30:59.572205 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828e950a_ac6a_45ed_abb7_b11753927f23.slice/crio-8c4343e1cf5ae07f2a27a5f81d036fd93eebf33ed3926276f0ee58909d99b8a1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828e950a_ac6a_45ed_abb7_b11753927f23.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:31:00.950218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:00.950176 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828e950a-ac6a-45ed-abb7-b11753927f23" path="/var/lib/kubelet/pods/828e950a-ac6a-45ed-abb7-b11753927f23/volumes" Nov 03 22:31:01.665185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.665127 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd"] Nov 03 22:31:01.666230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666206 4625 cpu_manager.go:414] "containerMap: removing container" podUID="828e950a-ac6a-45ed-abb7-b11753927f23" containerName="maas-api" Nov 03 22:31:01.666230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666228 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828e950a-ac6a-45ed-abb7-b11753927f23" containerName="maas-api" Nov 03 22:31:01.666420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666264 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:31:01.666420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666273 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:31:01.666420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666299 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:31:01.666420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666307 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:31:01.666648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666451 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e87482-394e-481d-993b-590fd0a6eb3e" containerName="cleanup" Nov 03 22:31:01.666648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666466 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="828e950a-ac6a-45ed-abb7-b11753927f23" containerName="maas-api" Nov 03 22:31:01.666648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666477 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f72d4d-599c-49ab-93fb-7b566b020708" containerName="cleanup" Nov 03 22:31:01.666648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.666489 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8934af8-8db7-4ce4-a899-7abdfef8a34f" containerName="cleanup" Nov 03 22:31:01.702783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.691982 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.702783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.697067 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd"] Nov 03 22:31:01.706833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.706345 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-281732-openshift-default-dockercfg-7xd5k\"" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.843867 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-istio-token\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.843918 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904859f4-8b39-48ed-8289-283eeedec8bd-istiod-ca-cert\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.843986 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904859f4-8b39-48ed-8289-283eeedec8bd-istio-podinfo\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.844115 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8lh\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-kube-api-access-7p8lh\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.844148 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-credential-socket\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.844199 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-data\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.844247 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-envoy\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.844332 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-socket\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.844745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.844467 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-certs\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.949776 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-envoy\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.949877 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-socket\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.950001 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-certs\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.950132 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-istio-token\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.950159 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904859f4-8b39-48ed-8289-283eeedec8bd-istiod-ca-cert\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.950223 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904859f4-8b39-48ed-8289-283eeedec8bd-istio-podinfo\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.950505 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8lh\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-kube-api-access-7p8lh\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.950586 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-credential-socket\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.950662 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-data\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.951760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.951405 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-data\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.953098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.953077 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-certs\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.967739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.957076 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-socket\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.972875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.970083 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-credential-socket\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.974020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.973991 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904859f4-8b39-48ed-8289-283eeedec8bd-istiod-ca-cert\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.988824 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.981604 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904859f4-8b39-48ed-8289-283eeedec8bd-istio-podinfo\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:01.988824 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:01.984063 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-envoy\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:02.013740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:02.011551 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8lh\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-kube-api-access-7p8lh\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:02.030240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:02.022299 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-istio-token\") pod \"e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:02.307810 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:31:02.306365 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:31:02.317552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:02.317521 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:02.901945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:02.901919 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd"] Nov 03 22:31:02.903598 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:31:02.903571 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904859f4_8b39_48ed_8289_283eeedec8bd.slice/crio-bb6d768bc3e7d541581fb0e274166faf5ad42482c116437839cddce647d27051 WatchSource:0}: Error finding container bb6d768bc3e7d541581fb0e274166faf5ad42482c116437839cddce647d27051: Status 404 returned error can't find the container with id bb6d768bc3e7d541581fb0e274166faf5ad42482c116437839cddce647d27051 Nov 03 22:31:02.906620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:02.906586 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:02.906751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:02.906655 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:02.906751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:02.906689 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:03.611575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:03.601848 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" event={"ID":"904859f4-8b39-48ed-8289-283eeedec8bd","Type":"ContainerStarted","Data":"4de94923a3b9b694292f435d1bae87e25d7b9594ef56e173cb54ed100c3f7f90"} Nov 03 22:31:03.611575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:03.601937 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" event={"ID":"904859f4-8b39-48ed-8289-283eeedec8bd","Type":"ContainerStarted","Data":"bb6d768bc3e7d541581fb0e274166faf5ad42482c116437839cddce647d27051"} Nov 03 22:31:03.690834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:03.690772 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podStartSLOduration=2.6907527890000003 podStartE2EDuration="2.690752789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:03.67639505 +0000 UTC m=+1085.105731047" watchObservedRunningTime="2026-11-03 22:31:03.690752789 +0000 UTC m=+1085.120088784" Nov 03 22:31:04.319739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:04.318359 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:04.333234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:04.333197 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:04.610985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:04.610949 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:04.612038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:04.612017 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:31:06.859757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:06.856296 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9"] Nov 03 22:31:06.859757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:06.856558 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" podUID="1b4ba330-3769-4b16-956f-817c57000c81" containerName="maas-api" containerID="cri-o://c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e" gracePeriod=30 Nov 03 22:31:07.577964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.576580 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:31:07.708809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.636419 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4ba330-3769-4b16-956f-817c57000c81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4ba330-3769-4b16-956f-817c57000c81-openshift-service-ca\") pod \"1b4ba330-3769-4b16-956f-817c57000c81\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " Nov 03 22:31:07.708809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.636601 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-maas-api-tls\") pod \"1b4ba330-3769-4b16-956f-817c57000c81\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " Nov 03 22:31:07.708809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.636711 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-metrics-certs\") pod \"1b4ba330-3769-4b16-956f-817c57000c81\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " Nov 03 22:31:07.708809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.636855 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4ba330-3769-4b16-956f-817c57000c81-kube-api-access-x7262\" (UniqueName: \"kubernetes.io/projected/1b4ba330-3769-4b16-956f-817c57000c81-kube-api-access-x7262\") pod \"1b4ba330-3769-4b16-956f-817c57000c81\" (UID: \"1b4ba330-3769-4b16-956f-817c57000c81\") " Nov 03 22:31:07.729735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.717404 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4ba330-3769-4b16-956f-817c57000c81-openshift-service-ca" pod "1b4ba330-3769-4b16-956f-817c57000c81" (UID: "1b4ba330-3769-4b16-956f-817c57000c81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:31:07.921257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.921224 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4ba330-3769-4b16-956f-817c57000c81-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:07.946383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.946335 4625 generic.go:356] "Generic (PLEG): container finished" podID="1b4ba330-3769-4b16-956f-817c57000c81" containerID="c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e" exitCode=0 Nov 03 22:31:07.984729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.946596 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" event={"ID":"1b4ba330-3769-4b16-956f-817c57000c81","Type":"ContainerDied","Data":"c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e"} Nov 03 22:31:07.984729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.969002 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" event={"ID":"1b4ba330-3769-4b16-956f-817c57000c81","Type":"ContainerDied","Data":"0a85463a406b28a1efdb1e9b07d10280951d3567fb46661099903c6b1bdf70e1"} Nov 03 22:31:07.984729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.969046 4625 scope.go:122] "RemoveContainer" containerID="c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e" Nov 03 22:31:07.984729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.946692 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9" Nov 03 22:31:07.984729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.970988 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4ba330-3769-4b16-956f-817c57000c81-kube-api-access-x7262" pod "1b4ba330-3769-4b16-956f-817c57000c81" (UID: "1b4ba330-3769-4b16-956f-817c57000c81"). InnerVolumeSpecName "kube-api-access-x7262". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:31:07.984729 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.975678 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-maas-api-tls" pod "1b4ba330-3769-4b16-956f-817c57000c81" (UID: "1b4ba330-3769-4b16-956f-817c57000c81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:31:08.007785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.986096 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74"] Nov 03 22:31:08.024749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.009087 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" podUID="bff85a6f-57be-4d67-b5f8-a6a630d33391" containerName="istio-proxy" containerID="cri-o://4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b" gracePeriod=30 Nov 03 22:31:08.024749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:07.993288 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-metrics-certs" pod "1b4ba330-3769-4b16-956f-817c57000c81" (UID: "1b4ba330-3769-4b16-956f-817c57000c81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:31:08.138737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.125212 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:08.138737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.125246 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4ba330-3769-4b16-956f-817c57000c81-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:08.138737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.125261 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7262\" (UniqueName: \"kubernetes.io/projected/1b4ba330-3769-4b16-956f-817c57000c81-kube-api-access-x7262\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:08.187523 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.168951 4625 scope.go:122] "RemoveContainer" containerID="c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e" Nov 03 22:31:08.188023 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:31:08.187898 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e\": container with ID starting with c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e not found: ID does not exist" containerID="c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e" Nov 03 22:31:08.188023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.187942 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e"} err="failed to get container status \"c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e\": rpc error: code = NotFound desc = could not find container \"c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e\": container with ID starting with c6b7e417e03b61d25142e4116840b402e1212afe4dbdf5a0885ac29763cef00e not found: ID does not exist" Nov 03 22:31:08.435641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.435603 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9"] Nov 03 22:31:08.440649 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.440617 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbe6e297-58f87f5665-ndrz9"] Nov 03 22:31:08.741678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.741583 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4ba330-3769-4b16-956f-817c57000c81" path="/var/lib/kubelet/pods/1b4ba330-3769-4b16-956f-817c57000c81/volumes" Nov 03 22:31:08.957653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.957601 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq"] Nov 03 22:31:08.958927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.958900 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1b4ba330-3769-4b16-956f-817c57000c81" containerName="maas-api" Nov 03 22:31:08.958927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.958927 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4ba330-3769-4b16-956f-817c57000c81" containerName="maas-api" Nov 03 22:31:08.959296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.959277 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4ba330-3769-4b16-956f-817c57000c81" containerName="maas-api" Nov 03 22:31:08.965728 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.965698 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:08.984882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.984854 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq"] Nov 03 22:31:08.988747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:08.987751 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-0847aa-openshift-default-dockercfg-dkbjf\"" Nov 03 22:31:09.026554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.026457 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.026554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.026530 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.026813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.026591 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.026813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.026633 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.026813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.026682 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.027118 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.027089 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.027216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.027159 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.027280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.027193 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdls2\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-kube-api-access-pdls2\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.027280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.027251 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.032529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.032502 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt"] Nov 03 22:31:09.037601 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.037576 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:09.063105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.062666 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-281732\"" Nov 03 22:31:09.065929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.065895 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt"] Nov 03 22:31:09.073436 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.072042 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-281732-dockercfg-v46gf\"" Nov 03 22:31:09.129269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129234 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129314 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129359 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129407 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129479 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129533 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129564 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdls2\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-kube-api-access-pdls2\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129645 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.129817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129762 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbvv\" (UniqueName: \"kubernetes.io/projected/d2360c3d-31be-4438-aa78-24489e87bb66-kube-api-access-fnbvv\") pod \"payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:09.130253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129912 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.130253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129919 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2360c3d-31be-4438-aa78-24489e87bb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:09.130253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.129920 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.130253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.130030 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.130253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.130107 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.130253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.130147 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.130540 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.130353 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.131837 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.131817 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.132262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.132245 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.160789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.160753 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdls2\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-kube-api-access-pdls2\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.161913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.161882 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.243750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.242034 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbvv\" (UniqueName: \"kubernetes.io/projected/d2360c3d-31be-4438-aa78-24489e87bb66-kube-api-access-fnbvv\") pod \"payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:09.243750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.242195 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2360c3d-31be-4438-aa78-24489e87bb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:09.243750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.243253 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2360c3d-31be-4438-aa78-24489e87bb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:09.315374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.315059 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:09.436827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.436754 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbvv\" (UniqueName: \"kubernetes.io/projected/d2360c3d-31be-4438-aa78-24489e87bb66-kube-api-access-fnbvv\") pod \"payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:09.690097 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:09.674127 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:10.214849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:10.214769 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq"] Nov 03 22:31:10.215834 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:31:10.215803 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544947a5_5b4d_4d74_a1f0_2d8c7b1a514d.slice/crio-0f3d8be0b9c758f8816abdb62a491ba521071fe580e6df8a37ea2603ba4afe4d WatchSource:0}: Error finding container 0f3d8be0b9c758f8816abdb62a491ba521071fe580e6df8a37ea2603ba4afe4d: Status 404 returned error can't find the container with id 0f3d8be0b9c758f8816abdb62a491ba521071fe580e6df8a37ea2603ba4afe4d Nov 03 22:31:10.218635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:10.218601 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:10.218735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:10.218668 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:10.218735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:10.218695 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:10.267091 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:10.265222 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt"] Nov 03 22:31:10.267091 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:31:10.266260 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2360c3d_31be_4438_aa78_24489e87bb66.slice/crio-aaca7b999ce63a39d2f6963e0cba5da623dd7c62f690a972143775f6bbe6f452 WatchSource:0}: Error finding container aaca7b999ce63a39d2f6963e0cba5da623dd7c62f690a972143775f6bbe6f452: Status 404 returned error can't find the container with id aaca7b999ce63a39d2f6963e0cba5da623dd7c62f690a972143775f6bbe6f452 Nov 03 22:31:11.027746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.027672 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" event={"ID":"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d","Type":"ContainerStarted","Data":"1376f0546248cd71ed2616cf361be767a8a0e40766b5c3a473e51b6b20a6d104"} Nov 03 22:31:11.027926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.027752 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" event={"ID":"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d","Type":"ContainerStarted","Data":"0f3d8be0b9c758f8816abdb62a491ba521071fe580e6df8a37ea2603ba4afe4d"} Nov 03 22:31:11.029582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.029553 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" event={"ID":"d2360c3d-31be-4438-aa78-24489e87bb66","Type":"ContainerStarted","Data":"f852378ad4498820bea3638803feabd078d3389d1001dafaa0376b89d0909fae"} Nov 03 22:31:11.029695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.029604 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" event={"ID":"d2360c3d-31be-4438-aa78-24489e87bb66","Type":"ContainerStarted","Data":"aaca7b999ce63a39d2f6963e0cba5da623dd7c62f690a972143775f6bbe6f452"} Nov 03 22:31:11.029695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.029627 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:11.185739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.177995 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" podStartSLOduration=2.177972632 podStartE2EDuration="2.177972632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:11.15915802 +0000 UTC m=+1092.588494013" watchObservedRunningTime="2026-11-03 22:31:11.177972632 +0000 UTC m=+1092.607308639" Nov 03 22:31:11.205032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.204974 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podStartSLOduration=3.204958737 podStartE2EDuration="3.204958737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:11.109161661 +0000 UTC m=+1092.538497654" watchObservedRunningTime="2026-11-03 22:31:11.204958737 +0000 UTC m=+1092.634294730" Nov 03 22:31:11.317199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.317118 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:11.330982 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.325845 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:11.567196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.567160 4625 patch_prober.go:29] interesting pod/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:31:11.567499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:11.567429 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" podUID="bff85a6f-57be-4d67-b5f8-a6a630d33391" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:31:12.039940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:12.039904 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:12.040965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:12.040940 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:31:13.345319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.345294 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:31:13.369079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369048 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-envoy\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369096 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-token\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-token\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369128 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-credential-socket\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369181 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-kube-api-access-x27g4\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-kube-api-access-x27g4\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369267 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-podinfo\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369298 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-data\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369381 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-certs\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369452 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-socket\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369579 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff85a6f-57be-4d67-b5f8-a6a630d33391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff85a6f-57be-4d67-b5f8-a6a630d33391-istiod-ca-cert\") pod \"bff85a6f-57be-4d67-b5f8-a6a630d33391\" (UID: \"bff85a6f-57be-4d67-b5f8-a6a630d33391\") " Nov 03 22:31:13.369699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369652 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-data" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:31:13.369699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369683 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-certs" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:31:13.369944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369921 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-socket" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:31:13.370029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.370008 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff85a6f-57be-4d67-b5f8-a6a630d33391-istiod-ca-cert" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:31:13.370652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.370626 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.370791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.370658 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.370791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.370672 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.370791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.370686 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff85a6f-57be-4d67-b5f8-a6a630d33391-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.370791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.369826 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-credential-socket" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:31:13.371667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.371629 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-envoy" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:31:13.372256 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.372229 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-kube-api-access-x27g4" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "kube-api-access-x27g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:31:13.372361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.372339 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-podinfo" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:31:13.373092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.373071 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-token" pod "bff85a6f-57be-4d67-b5f8-a6a630d33391" (UID: "bff85a6f-57be-4d67-b5f8-a6a630d33391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:31:13.473152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.473113 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.473152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.473149 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.473361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.473167 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff85a6f-57be-4d67-b5f8-a6a630d33391-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.473361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.473185 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27g4\" (UniqueName: \"kubernetes.io/projected/bff85a6f-57be-4d67-b5f8-a6a630d33391-kube-api-access-x27g4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:13.473361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:13.473199 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff85a6f-57be-4d67-b5f8-a6a630d33391-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:31:14.060075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.060038 4625 generic.go:356] "Generic (PLEG): container finished" podID="bff85a6f-57be-4d67-b5f8-a6a630d33391" containerID="4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b" exitCode=0 Nov 03 22:31:14.060275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.060089 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" Nov 03 22:31:14.060275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.060129 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" event={"ID":"bff85a6f-57be-4d67-b5f8-a6a630d33391","Type":"ContainerDied","Data":"4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b"} Nov 03 22:31:14.060275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.060193 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74" event={"ID":"bff85a6f-57be-4d67-b5f8-a6a630d33391","Type":"ContainerDied","Data":"120ccd2b8d218c1e468fb57ecb096b53d848201a0a6d8ba14e9acb0e5877dba0"} Nov 03 22:31:14.060275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.060259 4625 scope.go:122] "RemoveContainer" containerID="4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b" Nov 03 22:31:14.072831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.072801 4625 scope.go:122] "RemoveContainer" containerID="4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b" Nov 03 22:31:14.073162 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:31:14.073138 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b\": container with ID starting with 4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b not found: ID does not exist" containerID="4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b" Nov 03 22:31:14.073240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.073170 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b"} err="failed to get container status \"4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b\": rpc error: code = NotFound desc = could not find container \"4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b\": container with ID starting with 4c86347007771308b92387314124bcbc9788377e5224603fa7984e2e0272f85b not found: ID does not exist" Nov 03 22:31:14.123691 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.123659 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74"] Nov 03 22:31:14.145003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.144964 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cbe6e297-openshift-default-795c76bc85-vgf74"] Nov 03 22:31:14.730080 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:14.730047 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff85a6f-57be-4d67-b5f8-a6a630d33391" path="/var/lib/kubelet/pods/bff85a6f-57be-4d67-b5f8-a6a630d33391/volumes" Nov 03 22:31:15.316764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.316683 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng"] Nov 03 22:31:15.317531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.317513 4625 cpu_manager.go:414] "containerMap: removing container" podUID="bff85a6f-57be-4d67-b5f8-a6a630d33391" containerName="istio-proxy" Nov 03 22:31:15.317593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.317533 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff85a6f-57be-4d67-b5f8-a6a630d33391" containerName="istio-proxy" Nov 03 22:31:15.317659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.317650 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff85a6f-57be-4d67-b5f8-a6a630d33391" containerName="istio-proxy" Nov 03 22:31:15.323116 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.323092 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:15.337707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.337484 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-0847aa\"" Nov 03 22:31:15.340805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.340778 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng"] Nov 03 22:31:15.398187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.398148 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm"] Nov 03 22:31:15.422259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.422230 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:15.431395 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.424617 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm"] Nov 03 22:31:15.431395 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.425938 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4ab126-b50f-4673-8ff5-12444e47b637-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:15.431395 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.426066 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf7lr\" (UniqueName: \"kubernetes.io/projected/ac4ab126-b50f-4673-8ff5-12444e47b637-kube-api-access-nf7lr\") pod \"payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:15.473743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.449855 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-0847aa-dockercfg-gkkpq\"" Nov 03 22:31:15.604741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.599160 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghgq\" (UniqueName: \"kubernetes.io/projected/7fc247d5-97b3-4b2d-97e8-e08346346447-kube-api-access-9ghgq\") pod \"payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:15.604741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.599290 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4ab126-b50f-4673-8ff5-12444e47b637-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:15.604741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.599416 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf7lr\" (UniqueName: \"kubernetes.io/projected/ac4ab126-b50f-4673-8ff5-12444e47b637-kube-api-access-nf7lr\") pod \"payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:15.604741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.599563 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc247d5-97b3-4b2d-97e8-e08346346447-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:15.604741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.600564 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4ab126-b50f-4673-8ff5-12444e47b637-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:15.711805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.707363 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc247d5-97b3-4b2d-97e8-e08346346447-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:15.711805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.707533 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghgq\" (UniqueName: \"kubernetes.io/projected/7fc247d5-97b3-4b2d-97e8-e08346346447-kube-api-access-9ghgq\") pod \"payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:15.739784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.723422 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc247d5-97b3-4b2d-97e8-e08346346447-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:15.797552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.786890 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf7lr\" (UniqueName: \"kubernetes.io/projected/ac4ab126-b50f-4673-8ff5-12444e47b637-kube-api-access-nf7lr\") pod \"payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:15.857936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.857849 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghgq\" (UniqueName: \"kubernetes.io/projected/7fc247d5-97b3-4b2d-97e8-e08346346447-kube-api-access-9ghgq\") pod \"payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:15.951765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:15.946003 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:16.159394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:16.159048 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:16.505404 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:16.505374 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng"] Nov 03 22:31:16.506614 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:31:16.506591 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4ab126_b50f_4673_8ff5_12444e47b637.slice/crio-14d92705f74706016a4a63baa46a177739a48818617682849acb7cd49bd8fcbc WatchSource:0}: Error finding container 14d92705f74706016a4a63baa46a177739a48818617682849acb7cd49bd8fcbc: Status 404 returned error can't find the container with id 14d92705f74706016a4a63baa46a177739a48818617682849acb7cd49bd8fcbc Nov 03 22:31:16.609286 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:31:16.609255 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc247d5_97b3_4b2d_97e8_e08346346447.slice/crio-150bca91491b8028fc7ff6b3ef12dfacc0d46355ed48e36fc79eefcd214d8d15 WatchSource:0}: Error finding container 150bca91491b8028fc7ff6b3ef12dfacc0d46355ed48e36fc79eefcd214d8d15: Status 404 returned error can't find the container with id 150bca91491b8028fc7ff6b3ef12dfacc0d46355ed48e36fc79eefcd214d8d15 Nov 03 22:31:16.614273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:16.614242 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm"] Nov 03 22:31:17.097162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.097125 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" event={"ID":"7fc247d5-97b3-4b2d-97e8-e08346346447","Type":"ContainerStarted","Data":"44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb"} Nov 03 22:31:17.097604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.097181 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" event={"ID":"7fc247d5-97b3-4b2d-97e8-e08346346447","Type":"ContainerStarted","Data":"150bca91491b8028fc7ff6b3ef12dfacc0d46355ed48e36fc79eefcd214d8d15"} Nov 03 22:31:17.097604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.097211 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:17.098974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.098946 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" event={"ID":"ac4ab126-b50f-4673-8ff5-12444e47b637","Type":"ContainerStarted","Data":"40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506"} Nov 03 22:31:17.099099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.098990 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" event={"ID":"ac4ab126-b50f-4673-8ff5-12444e47b637","Type":"ContainerStarted","Data":"14d92705f74706016a4a63baa46a177739a48818617682849acb7cd49bd8fcbc"} Nov 03 22:31:17.099099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.099012 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:17.187647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.187589 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" podStartSLOduration=2.187573632 podStartE2EDuration="2.187573632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:17.18468775 +0000 UTC m=+1098.614023746" watchObservedRunningTime="2026-11-03 22:31:17.187573632 +0000 UTC m=+1098.616909622" Nov 03 22:31:17.239428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:17.239374 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" podStartSLOduration=2.239357823 podStartE2EDuration="2.239357823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:17.237033088 +0000 UTC m=+1098.666369081" watchObservedRunningTime="2026-11-03 22:31:17.239357823 +0000 UTC m=+1098.668693812" Nov 03 22:31:20.859330 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:20.859282 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp"] Nov 03 22:31:20.865406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:20.865378 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:20.890832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:20.886027 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-281732-serving-cert\"" Nov 03 22:31:20.908167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:20.908133 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp"] Nov 03 22:31:21.024345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.024293 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.026430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.026393 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.026796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.026772 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed4df38-d21b-4b49-a0c8-076f51459320-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.027199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.027177 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpf8\" (UniqueName: \"kubernetes.io/projected/4ed4df38-d21b-4b49-a0c8-076f51459320-kube-api-access-gfpf8\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.208787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.206018 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.208787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.206180 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed4df38-d21b-4b49-a0c8-076f51459320-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.208787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.206491 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpf8\" (UniqueName: \"kubernetes.io/projected/4ed4df38-d21b-4b49-a0c8-076f51459320-kube-api-access-gfpf8\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.208787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.206608 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.208787 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:31:21.207011 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-serving-cert: secret "maas-api-e2e-shared-a-w4-281732-serving-cert" not found Nov 03 22:31:21.208787 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:31:21.207114 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls podName:4ed4df38-d21b-4b49-a0c8-076f51459320 nodeName:}" failed. No retries permitted until 2026-11-03 22:31:21.707088244 +0000 UTC m=+1103.136424228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls") pod "maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" (UID: "4ed4df38-d21b-4b49-a0c8-076f51459320") : secret "maas-api-e2e-shared-a-w4-281732-serving-cert" not found Nov 03 22:31:21.208787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.208385 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed4df38-d21b-4b49-a0c8-076f51459320-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.390347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.389976 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.411555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.411521 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpf8\" (UniqueName: \"kubernetes.io/projected/4ed4df38-d21b-4b49-a0c8-076f51459320-kube-api-access-gfpf8\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.791271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.790592 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.795024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.794996 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:21.817746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:21.796669 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:22.101851 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:22.095317 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 22:31:23.296800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:23.285376 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:23.296800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:23.285477 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:23.368951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:23.368915 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:23.369298 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:23.369250 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:23.709562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:23.709287 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp"] Nov 03 22:31:24.211111 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:24.209371 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" event={"ID":"4ed4df38-d21b-4b49-a0c8-076f51459320","Type":"ContainerStarted","Data":"8140c48b6dfb56d0de2b5e05e8ff54ab88c0d6f05d6f9839ae7a4e9c174b98cc"} Nov 03 22:31:24.406949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:24.405038 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:31:24.624547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:24.619891 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:25.096795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:25.092842 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:25.096795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:25.092907 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:25.284741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:25.279768 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" event={"ID":"4ed4df38-d21b-4b49-a0c8-076f51459320","Type":"ContainerStarted","Data":"f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d"} Nov 03 22:31:25.284741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:25.279874 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:25.342582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:25.342513 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" podStartSLOduration=4.406113168 podStartE2EDuration="5.342491157s" totalImagesPullingTime="936.377989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:25.326887165 +0000 UTC m=+1106.756223156" watchObservedRunningTime="2026-11-03 22:31:25.342491157 +0000 UTC m=+1106.771827151" Nov 03 22:31:28.110270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:28.110243 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 22:31:28.110782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:28.110324 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 22:31:29.242611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.242564 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2"] Nov 03 22:31:29.247810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.247785 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.264509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.264478 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-0847aa-serving-cert\"" Nov 03 22:31:29.283219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.283181 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2"] Nov 03 22:31:29.379051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.379009 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2sk\" (UniqueName: \"kubernetes.io/projected/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-kube-api-access-mb2sk\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.379301 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.379278 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.379397 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.379350 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.379504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.379477 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.481123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.481086 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2sk\" (UniqueName: \"kubernetes.io/projected/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-kube-api-access-mb2sk\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.481338 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.481267 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.481338 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.481315 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.481462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.481366 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.482137 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.482102 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.483900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.483878 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.484003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.483973 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.548796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.546872 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2sk\" (UniqueName: \"kubernetes.io/projected/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-kube-api-access-mb2sk\") pod \"maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:29.650740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:29.635798 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:30.200680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:30.200391 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2"] Nov 03 22:31:30.366518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:30.366474 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" event={"ID":"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3","Type":"ContainerStarted","Data":"b3a9e211e68c7662639da0fd2ac092c03628273bdab585f5f079a8e9c6832147"} Nov 03 22:31:31.335362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:31.335335 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 22:31:31.378971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:31.378930 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" event={"ID":"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3","Type":"ContainerStarted","Data":"b8f44f995d672a7e100ca5c49dd7e340fdb0ca622636c973c336e453e03c5d41"} Nov 03 22:31:31.379437 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:31.379044 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:31.439738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:31.439677 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" podStartSLOduration=1.819890419 podStartE2EDuration="2.439662608s" totalImagesPullingTime="619.772189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:31.43311029 +0000 UTC m=+1112.862446284" watchObservedRunningTime="2026-11-03 22:31:31.439662608 +0000 UTC m=+1112.868998599" Nov 03 22:31:32.187903 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:31:32.187866 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:31:37.411739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:37.410837 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 22:31:42.536583 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.536498 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb"] Nov 03 22:31:42.554739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.547929 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.586201 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.585592 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb"] Nov 03 22:31:42.588062 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.588035 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-04988c-openshift-default-dockercfg-6kp47\"" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.700443 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxsnp\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-kube-api-access-rxsnp\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.700553 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.700711 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-data\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.700761 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-token\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.700930 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9903686-a262-41cd-b4fd-2bf5936cddca-istiod-ca-cert\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.700971 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-credential-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.701012 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-certs\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.701061 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-envoy\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.701746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.701098 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-podinfo\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.827791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.824473 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9903686-a262-41cd-b4fd-2bf5936cddca-istiod-ca-cert\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.834002 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.825337 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9903686-a262-41cd-b4fd-2bf5936cddca-istiod-ca-cert\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.852739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.834574 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-credential-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.853096 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.853048 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-credential-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.853330 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.853310 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-certs\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.853488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.853473 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-envoy\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.853631 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.853617 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-podinfo\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.853806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.853787 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxsnp\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-kube-api-access-rxsnp\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.854006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.853988 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.854288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.854273 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-data\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.854477 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.854453 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-token\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.855408 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.855386 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-certs\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.862754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.857061 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-data\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.862754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.857676 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.898736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.878705 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-envoy\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.898736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.879235 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-podinfo\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.898736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.890195 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-token\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:42.901127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:42.899557 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxsnp\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-kube-api-access-rxsnp\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:43.201790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:43.197126 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:43.660836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:43.660803 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb"] Nov 03 22:31:43.661234 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:31:43.660995 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9903686_a262_41cd_b4fd_2bf5936cddca.slice/crio-a2b3eea1630fc386aaf27e6bccb31323f58299ad864e4f62591e002ca32ef048 WatchSource:0}: Error finding container a2b3eea1630fc386aaf27e6bccb31323f58299ad864e4f62591e002ca32ef048: Status 404 returned error can't find the container with id a2b3eea1630fc386aaf27e6bccb31323f58299ad864e4f62591e002ca32ef048 Nov 03 22:31:43.663938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:43.663904 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:43.664056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:43.663971 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:43.664056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:43.664018 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:44.550265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:44.550222 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" event={"ID":"f9903686-a262-41cd-b4fd-2bf5936cddca","Type":"ContainerStarted","Data":"9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc"} Nov 03 22:31:44.550474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:44.550296 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" event={"ID":"f9903686-a262-41cd-b4fd-2bf5936cddca","Type":"ContainerStarted","Data":"a2b3eea1630fc386aaf27e6bccb31323f58299ad864e4f62591e002ca32ef048"} Nov 03 22:31:44.619325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:44.619274 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podStartSLOduration=2.619255846 podStartE2EDuration="2.619255846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:44.604887942 +0000 UTC m=+1126.034223936" watchObservedRunningTime="2026-11-03 22:31:44.619255846 +0000 UTC m=+1126.048591838" Nov 03 22:31:45.198508 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:45.198470 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:45.204936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:45.204905 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:45.559452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:45.559359 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:45.560476 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:45.560420 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:31:55.110691 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.110599 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r"] Nov 03 22:31:55.118183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.118151 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.145834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.145805 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a940d5-openshift-default-dockercfg-9b7lq\"" Nov 03 22:31:55.154645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.154609 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r"] Nov 03 22:31:55.195200 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.195155 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d1897-361b-493e-a6ce-081d556893a7-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.195542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.195521 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgm4w\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-kube-api-access-dgm4w\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.195757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.195736 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.195979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.195962 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d1897-361b-493e-a6ce-081d556893a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.196190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.196174 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.196478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.196461 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.196663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.196649 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.196983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.196963 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.197121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.197106 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299237 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299473 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299293 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299473 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299335 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299473 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299363 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299473 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299444 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d1897-361b-493e-a6ce-081d556893a7-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299475 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgm4w\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-kube-api-access-dgm4w\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299541 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299585 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d1897-361b-493e-a6ce-081d556893a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.299704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299683 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.300145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299745 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.300145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299810 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.300145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.299865 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.300314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.300220 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.300626 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.300602 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d1897-361b-493e-a6ce-081d556893a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.302378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.302355 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d1897-361b-493e-a6ce-081d556893a7-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.302499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.302451 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.326692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.326651 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.332619 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.332585 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgm4w\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-kube-api-access-dgm4w\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:55.480792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:55.477206 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:56.390050 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:56.379565 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r"] Nov 03 22:31:56.418933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:56.410034 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:56.418933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:56.410134 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:56.418933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:56.410179 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:31:56.974694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:56.974601 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" event={"ID":"652d1897-361b-493e-a6ce-081d556893a7","Type":"ContainerStarted","Data":"450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8"} Nov 03 22:31:56.974694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:56.974656 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" event={"ID":"652d1897-361b-493e-a6ce-081d556893a7","Type":"ContainerStarted","Data":"340178512ce4cb1505968368d61fdffadf3ebd40857f875ee7f154654f192de5"} Nov 03 22:31:57.487695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:57.484500 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:57.517114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:57.517075 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:57.569857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:57.566681 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podStartSLOduration=2.566660423 podStartE2EDuration="2.566660423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:57.036109846 +0000 UTC m=+1138.465445841" watchObservedRunningTime="2026-11-03 22:31:57.566660423 +0000 UTC m=+1138.995996421" Nov 03 22:31:57.991070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:57.991034 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:31:57.992255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:31:57.992229 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:32:01.615736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:01.615193 4625 scope.go:122] "RemoveContainer" containerID="33f83aa8aea956c3bc0dec85fd7f27c5041a5230f8a51b5a2b9b6dd6b6f15799" Nov 03 22:32:01.716453 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:01.716430 4625 scope.go:122] "RemoveContainer" containerID="d24d2360725745d6200bac1f34ff89b69b933c971856e92869f08da8ae291ba9" Nov 03 22:32:01.731282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:01.731254 4625 scope.go:122] "RemoveContainer" containerID="0fdbf9b9dcd6b5b2940dc9059e80caf6bc25f08c2eb8061913d26135c29776bd" Nov 03 22:32:01.865003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:01.860166 4625 scope.go:122] "RemoveContainer" containerID="0aba23a7d2d31502bd0579702588e0adf506ccbb8e25dbcea12bf9963f16125a" Nov 03 22:32:02.631757 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:02.608885 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:32:05.652933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.652887 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:05.653482 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.652954 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:05.892896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.892863 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6"] Nov 03 22:32:05.901101 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.901057 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:05.916880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.916239 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6"] Nov 03 22:32:05.918094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.918069 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-04988c-serving-cert\"" Nov 03 22:32:05.931758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.931230 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70a8fa5-192c-486d-825f-d45c9342c6ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:05.931758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.931299 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:05.931758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.931341 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:05.931758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:05.931456 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhx7\" (UniqueName: \"kubernetes.io/projected/f70a8fa5-192c-486d-825f-d45c9342c6ca-kube-api-access-snhx7\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.034034 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.033993 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhx7\" (UniqueName: \"kubernetes.io/projected/f70a8fa5-192c-486d-825f-d45c9342c6ca-kube-api-access-snhx7\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.034792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.034762 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70a8fa5-192c-486d-825f-d45c9342c6ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.034999 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.034979 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.035127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.035111 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.035705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.035681 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70a8fa5-192c-486d-825f-d45c9342c6ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.036858 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:06.036836 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-serving-cert: secret "maas-api-e2e-shared-b-w4-04988c-serving-cert" not found Nov 03 22:32:06.036958 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:06.036920 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls podName:f70a8fa5-192c-486d-825f-d45c9342c6ca nodeName:}" failed. No retries permitted until 2026-11-03 22:32:06.536898879 +0000 UTC m=+1147.966234865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls") pod "maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" (UID: "f70a8fa5-192c-486d-825f-d45c9342c6ca") : secret "maas-api-e2e-shared-b-w4-04988c-serving-cert" not found Nov 03 22:32:06.040829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.040807 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.102042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.075450 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhx7\" (UniqueName: \"kubernetes.io/projected/f70a8fa5-192c-486d-825f-d45c9342c6ca-kube-api-access-snhx7\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.557238 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.557191 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.578770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.561755 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:06.835793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:06.828034 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:07.664887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:07.650732 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6"] Nov 03 22:32:07.855787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:07.855587 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" event={"ID":"f70a8fa5-192c-486d-825f-d45c9342c6ca","Type":"ContainerStarted","Data":"799573b6f8759954f254f1de5197ff08d85367de3037290b020be7e8e23d1e8f"} Nov 03 22:32:08.867163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:08.867129 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" event={"ID":"f70a8fa5-192c-486d-825f-d45c9342c6ca","Type":"ContainerStarted","Data":"87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c"} Nov 03 22:32:08.867651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:08.867358 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:08.912217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:08.912164 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" podStartSLOduration=3.119853446 podStartE2EDuration="3.91214979s" totalImagesPullingTime="792.296344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:08.908135129 +0000 UTC m=+1150.337471122" watchObservedRunningTime="2026-11-03 22:32:08.91214979 +0000 UTC m=+1150.341485831" Nov 03 22:32:10.931798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.930964 4625 generic.go:356] "Generic (PLEG): container finished" podID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerID="1376f0546248cd71ed2616cf361be767a8a0e40766b5c3a473e51b6b20a6d104" exitCode=0 Nov 03 22:32:10.931798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.931166 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" event={"ID":"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d","Type":"ContainerDied","Data":"1376f0546248cd71ed2616cf361be767a8a0e40766b5c3a473e51b6b20a6d104"} Nov 03 22:32:10.947844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.945745 4625 scope.go:122] "RemoveContainer" containerID="1376f0546248cd71ed2616cf361be767a8a0e40766b5c3a473e51b6b20a6d104" Nov 03 22:32:10.961745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.959755 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:10.961745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.959905 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:10.961745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.959969 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:10.971043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.970835 4625 generic.go:356] "Generic (PLEG): container finished" podID="904859f4-8b39-48ed-8289-283eeedec8bd" containerID="4de94923a3b9b694292f435d1bae87e25d7b9594ef56e173cb54ed100c3f7f90" exitCode=0 Nov 03 22:32:10.971043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.970904 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" event={"ID":"904859f4-8b39-48ed-8289-283eeedec8bd","Type":"ContainerDied","Data":"4de94923a3b9b694292f435d1bae87e25d7b9594ef56e173cb54ed100c3f7f90"} Nov 03 22:32:10.971412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:10.971393 4625 scope.go:122] "RemoveContainer" containerID="4de94923a3b9b694292f435d1bae87e25d7b9594ef56e173cb54ed100c3f7f90" Nov 03 22:32:11.016527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:11.016432 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:11.016877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:11.016851 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:11.017043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:11.017026 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:12.003787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.003750 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" event={"ID":"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d","Type":"ContainerStarted","Data":"508f43fe2c469e18979c12ccf7f6c3e0666ac5a3d27bc5caaa82a06038f4bb06"} Nov 03 22:32:12.005704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.005676 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" event={"ID":"904859f4-8b39-48ed-8289-283eeedec8bd","Type":"ContainerStarted","Data":"fa58ea476a64a128da80ceede5cb7b009a07c3f84ce3381bd8bae23892034f85"} Nov 03 22:32:12.055749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.040827 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:32:12.316328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.316233 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:32:12.317230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.317209 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:32:12.317297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.317256 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 22:32:12.318355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.318330 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:32:12.318536 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.318516 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:32:12.318611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:12.318558 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 22:32:13.316744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:13.316677 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:32:13.317169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:13.316771 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 22:32:13.319114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:13.319084 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:32:13.319232 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:13.319142 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 22:32:14.329741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:14.322348 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:32:14.329741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:14.322405 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 22:32:14.329741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:14.322861 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:32:14.329741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:14.322898 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 22:32:14.592011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:14.591974 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn"] Nov 03 22:32:14.602410 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:14.602382 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z"] Nov 03 22:32:15.084793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:15.016000 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 22:32:15.316887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:15.316850 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:32:15.317045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:15.316932 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 22:32:15.318206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:15.318182 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:32:15.318314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:15.318221 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 22:32:15.637737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:15.637178 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv"] Nov 03 22:32:15.637737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:15.637438 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" podUID="5f24a2a5-4268-4bd4-b505-b5aed592b177" containerName="maas-api" containerID="cri-o://5afcc407b0b4a9ce67cc129ac139842002ed1451a95aaa40212d22791a0ef78c" gracePeriod=30 Nov 03 22:32:16.258740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:16.254025 4625 generic.go:356] "Generic (PLEG): container finished" podID="5f24a2a5-4268-4bd4-b505-b5aed592b177" containerID="5afcc407b0b4a9ce67cc129ac139842002ed1451a95aaa40212d22791a0ef78c" exitCode=0 Nov 03 22:32:16.258740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:16.254230 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" event={"ID":"5f24a2a5-4268-4bd4-b505-b5aed592b177","Type":"ContainerDied","Data":"5afcc407b0b4a9ce67cc129ac139842002ed1451a95aaa40212d22791a0ef78c"} Nov 03 22:32:16.317709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:16.317675 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:32:16.318031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:16.317999 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 22:32:16.326761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:16.318480 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:32:16.327676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:16.327647 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 22:32:17.119150 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.119119 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:32:17.297228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.297131 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" event={"ID":"5f24a2a5-4268-4bd4-b505-b5aed592b177","Type":"ContainerDied","Data":"c6b42aafd383675b940c7263318e3550e7bee7f0237675cb49e123fe09399902"} Nov 03 22:32:17.297473 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.297453 4625 scope.go:122] "RemoveContainer" containerID="5afcc407b0b4a9ce67cc129ac139842002ed1451a95aaa40212d22791a0ef78c" Nov 03 22:32:17.297813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.297794 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv" Nov 03 22:32:17.343423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.327993 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:32:17.343423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.328066 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 22:32:17.343423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.328779 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:32:17.343423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.328817 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 22:32:17.343423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.330936 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-maas-api-tls\") pod \"5f24a2a5-4268-4bd4-b505-b5aed592b177\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " Nov 03 22:32:17.402743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.393883 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-metrics-certs\") pod \"5f24a2a5-4268-4bd4-b505-b5aed592b177\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " Nov 03 22:32:17.402743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.394021 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f24a2a5-4268-4bd4-b505-b5aed592b177-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f24a2a5-4268-4bd4-b505-b5aed592b177-openshift-service-ca\") pod \"5f24a2a5-4268-4bd4-b505-b5aed592b177\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " Nov 03 22:32:17.402743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.394092 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f24a2a5-4268-4bd4-b505-b5aed592b177-kube-api-access-bqvlr\" (UniqueName: \"kubernetes.io/projected/5f24a2a5-4268-4bd4-b505-b5aed592b177-kube-api-access-bqvlr\") pod \"5f24a2a5-4268-4bd4-b505-b5aed592b177\" (UID: \"5f24a2a5-4268-4bd4-b505-b5aed592b177\") " Nov 03 22:32:17.404530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.403233 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f24a2a5-4268-4bd4-b505-b5aed592b177-openshift-service-ca" pod "5f24a2a5-4268-4bd4-b505-b5aed592b177" (UID: "5f24a2a5-4268-4bd4-b505-b5aed592b177"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:17.405420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.405391 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-maas-api-tls" pod "5f24a2a5-4268-4bd4-b505-b5aed592b177" (UID: "5f24a2a5-4268-4bd4-b505-b5aed592b177"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:17.411980 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.411939 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-metrics-certs" pod "5f24a2a5-4268-4bd4-b505-b5aed592b177" (UID: "5f24a2a5-4268-4bd4-b505-b5aed592b177"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:17.412570 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.412532 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f24a2a5-4268-4bd4-b505-b5aed592b177-kube-api-access-bqvlr" pod "5f24a2a5-4268-4bd4-b505-b5aed592b177" (UID: "5f24a2a5-4268-4bd4-b505-b5aed592b177"). InnerVolumeSpecName "kube-api-access-bqvlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:17.496621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.496586 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:17.496621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.496617 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f24a2a5-4268-4bd4-b505-b5aed592b177-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:17.496621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.496626 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqvlr\" (UniqueName: \"kubernetes.io/projected/5f24a2a5-4268-4bd4-b505-b5aed592b177-kube-api-access-bqvlr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:17.496885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.496635 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f24a2a5-4268-4bd4-b505-b5aed592b177-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:17.682816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.682777 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv"] Nov 03 22:32:17.694792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:17.694679 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-53879c92-64786dd47b-z27zv"] Nov 03 22:32:18.198819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.198772 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7"] Nov 03 22:32:18.201641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.201610 4625 cpu_manager.go:414] "containerMap: removing container" podUID="5f24a2a5-4268-4bd4-b505-b5aed592b177" containerName="maas-api" Nov 03 22:32:18.201898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.201882 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f24a2a5-4268-4bd4-b505-b5aed592b177" containerName="maas-api" Nov 03 22:32:18.202187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.202174 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f24a2a5-4268-4bd4-b505-b5aed592b177" containerName="maas-api" Nov 03 22:32:18.210822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.208097 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.212524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.212491 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7"] Nov 03 22:32:18.226954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.226590 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-a940d5-serving-cert\"" Nov 03 22:32:18.318228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.316454 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:32:18.318228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.316515 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 22:32:18.318498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.318365 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:32:18.318498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.318419 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 22:32:18.325816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.325780 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxjw\" (UniqueName: \"kubernetes.io/projected/676907ab-2cf3-4909-810b-848642fe6bb7-kube-api-access-7cxjw\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.325955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.325855 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.326063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.326024 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.326211 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.326146 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676907ab-2cf3-4909-810b-848642fe6bb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.456362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.456257 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676907ab-2cf3-4909-810b-848642fe6bb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.457724 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.457616 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676907ab-2cf3-4909-810b-848642fe6bb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.458628 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.458201 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxjw\" (UniqueName: \"kubernetes.io/projected/676907ab-2cf3-4909-810b-848642fe6bb7-kube-api-access-7cxjw\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.458852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.458831 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.459216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.459195 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.459921 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:18.459031 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-serving-cert: secret "maas-api-e2e-shared-b-w0-a940d5-serving-cert" not found Nov 03 22:32:18.460121 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:18.460107 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls podName:676907ab-2cf3-4909-810b-848642fe6bb7 nodeName:}" failed. No retries permitted until 2026-11-03 22:32:18.960079443 +0000 UTC m=+1160.389415434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls") pod "maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" (UID: "676907ab-2cf3-4909-810b-848642fe6bb7") : secret "maas-api-e2e-shared-b-w0-a940d5-serving-cert" not found Nov 03 22:32:18.463471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.463423 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.558270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.549538 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxjw\" (UniqueName: \"kubernetes.io/projected/676907ab-2cf3-4909-810b-848642fe6bb7-kube-api-access-7cxjw\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:18.733454 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.733385 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f24a2a5-4268-4bd4-b505-b5aed592b177" path="/var/lib/kubelet/pods/5f24a2a5-4268-4bd4-b505-b5aed592b177/volumes" Nov 03 22:32:18.978937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:18.977569 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:19.013861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.012397 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:19.192406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.192107 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:19.354155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.354114 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:32:19.371228 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.371196 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:32:19.376124 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.375458 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:32:19.389740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.387859 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 22:32:19.481239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.481209 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:32:19.592516 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.592480 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" podUID="5fd44272-3cab-46a1-92d0-e3a5849156d8" containerName="payload-processing" containerID="cri-o://4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5" gracePeriod=25 Nov 03 22:32:19.604357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.603320 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" podUID="3e6689e8-ec4d-4b52-a443-e882316b71f8" containerName="payload-pre-processing" containerID="cri-o://92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8" gracePeriod=25 Nov 03 22:32:19.862030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:19.860795 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7"] Nov 03 22:32:20.191952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.191919 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:32:20.253411 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.253309 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd44272-3cab-46a1-92d0-e3a5849156d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd44272-3cab-46a1-92d0-e3a5849156d8-plugins-config-volume\") pod \"5fd44272-3cab-46a1-92d0-e3a5849156d8\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " Nov 03 22:32:20.253615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.253522 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd44272-3cab-46a1-92d0-e3a5849156d8-kube-api-access-glpln\" (UniqueName: \"kubernetes.io/projected/5fd44272-3cab-46a1-92d0-e3a5849156d8-kube-api-access-glpln\") pod \"5fd44272-3cab-46a1-92d0-e3a5849156d8\" (UID: \"5fd44272-3cab-46a1-92d0-e3a5849156d8\") " Nov 03 22:32:20.254157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.254109 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd44272-3cab-46a1-92d0-e3a5849156d8-plugins-config-volume" pod "5fd44272-3cab-46a1-92d0-e3a5849156d8" (UID: "5fd44272-3cab-46a1-92d0-e3a5849156d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:20.254608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.254583 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd44272-3cab-46a1-92d0-e3a5849156d8-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:20.256136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.256113 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd44272-3cab-46a1-92d0-e3a5849156d8-kube-api-access-glpln" pod "5fd44272-3cab-46a1-92d0-e3a5849156d8" (UID: "5fd44272-3cab-46a1-92d0-e3a5849156d8"). InnerVolumeSpecName "kube-api-access-glpln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:20.268527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.268505 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:32:20.355695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.355661 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6689e8-ec4d-4b52-a443-e882316b71f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6689e8-ec4d-4b52-a443-e882316b71f8-plugins-config-volume\") pod \"3e6689e8-ec4d-4b52-a443-e882316b71f8\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " Nov 03 22:32:20.356169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.355891 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6689e8-ec4d-4b52-a443-e882316b71f8-kube-api-access-s24k7\" (UniqueName: \"kubernetes.io/projected/3e6689e8-ec4d-4b52-a443-e882316b71f8-kube-api-access-s24k7\") pod \"3e6689e8-ec4d-4b52-a443-e882316b71f8\" (UID: \"3e6689e8-ec4d-4b52-a443-e882316b71f8\") " Nov 03 22:32:20.356169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.356098 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6689e8-ec4d-4b52-a443-e882316b71f8-plugins-config-volume" pod "3e6689e8-ec4d-4b52-a443-e882316b71f8" (UID: "3e6689e8-ec4d-4b52-a443-e882316b71f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:20.356447 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.356430 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glpln\" (UniqueName: \"kubernetes.io/projected/5fd44272-3cab-46a1-92d0-e3a5849156d8-kube-api-access-glpln\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:20.356494 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.356450 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6689e8-ec4d-4b52-a443-e882316b71f8-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:20.358231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.358205 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6689e8-ec4d-4b52-a443-e882316b71f8-kube-api-access-s24k7" pod "3e6689e8-ec4d-4b52-a443-e882316b71f8" (UID: "3e6689e8-ec4d-4b52-a443-e882316b71f8"). InnerVolumeSpecName "kube-api-access-s24k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:20.467741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.461069 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s24k7\" (UniqueName: \"kubernetes.io/projected/3e6689e8-ec4d-4b52-a443-e882316b71f8-kube-api-access-s24k7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:20.661799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.661515 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" event={"ID":"676907ab-2cf3-4909-810b-848642fe6bb7","Type":"ContainerStarted","Data":"6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74"} Nov 03 22:32:20.661799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.661590 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" event={"ID":"676907ab-2cf3-4909-810b-848642fe6bb7","Type":"ContainerStarted","Data":"684f2b32f4622bda3929db2e86fcf4843fced7185736ea1e703a98f6efcadf22"} Nov 03 22:32:20.673741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.663031 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:20.678141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.676875 4625 generic.go:356] "Generic (PLEG): container finished" podID="3e6689e8-ec4d-4b52-a443-e882316b71f8" containerID="92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8" exitCode=0 Nov 03 22:32:20.678141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.677112 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" event={"ID":"3e6689e8-ec4d-4b52-a443-e882316b71f8","Type":"ContainerDied","Data":"92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8"} Nov 03 22:32:20.678141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.677173 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" event={"ID":"3e6689e8-ec4d-4b52-a443-e882316b71f8","Type":"ContainerDied","Data":"a0011a9d2da2aae49aca2de26befd4d0ca3e411af5b9c157830b9bd4018d6582"} Nov 03 22:32:20.678141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.677204 4625 scope.go:122] "RemoveContainer" containerID="92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8" Nov 03 22:32:20.678141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.677363 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z" Nov 03 22:32:20.726730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.726682 4625 generic.go:356] "Generic (PLEG): container finished" podID="5fd44272-3cab-46a1-92d0-e3a5849156d8" containerID="4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5" exitCode=0 Nov 03 22:32:20.728026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.728004 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" Nov 03 22:32:20.728599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.728483 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" event={"ID":"5fd44272-3cab-46a1-92d0-e3a5849156d8","Type":"ContainerDied","Data":"4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5"} Nov 03 22:32:20.728808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.728780 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn" event={"ID":"5fd44272-3cab-46a1-92d0-e3a5849156d8","Type":"ContainerDied","Data":"17f00903c0315ac77a341c5647b52233f745ac925212a7bc1f194a9c035f6fbe"} Nov 03 22:32:20.845601 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.845549 4625 scope.go:122] "RemoveContainer" containerID="92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8" Nov 03 22:32:20.846168 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:20.846138 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8\": container with ID starting with 92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8 not found: ID does not exist" containerID="92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8" Nov 03 22:32:20.846262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.846192 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8"} err="failed to get container status \"92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8\": rpc error: code = NotFound desc = could not find container \"92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8\": container with ID starting with 92f76238ab554b6ab2cb49c0f2a6aada4c08aa480a970820b3703ceb9d6ae2d8 not found: ID does not exist" Nov 03 22:32:20.846262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.846224 4625 scope.go:122] "RemoveContainer" containerID="4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5" Nov 03 22:32:20.880566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:20.880411 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" podStartSLOduration=2.537278857 podStartE2EDuration="2.880391588s" totalImagesPullingTime="343.112731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:20.725309299 +0000 UTC m=+1162.154645305" watchObservedRunningTime="2026-11-03 22:32:20.880391588 +0000 UTC m=+1162.309727583" Nov 03 22:32:21.075877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:21.033329 4625 scope.go:122] "RemoveContainer" containerID="4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5" Nov 03 22:32:21.076933 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:21.076894 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5\": container with ID starting with 4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5 not found: ID does not exist" containerID="4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5" Nov 03 22:32:21.077084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:21.076946 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5"} err="failed to get container status \"4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5\": rpc error: code = NotFound desc = could not find container \"4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5\": container with ID starting with 4e681e1c5578b9b0afb677fef3a9437b18703b55ee933c13386b337006f94be5 not found: ID does not exist" Nov 03 22:32:21.104101 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:21.104062 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6689e8_ec4d_4b52_a443_e882316b71f8.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:32:21.117731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:21.117687 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z"] Nov 03 22:32:21.122386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:21.122361 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53879c92-668d9d8855-74s4z"] Nov 03 22:32:21.152578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:21.152542 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn"] Nov 03 22:32:21.157285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:21.157256 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53879c92-6bf559b69b-plpwn"] Nov 03 22:32:22.730656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:22.730625 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6689e8-ec4d-4b52-a443-e882316b71f8" path="/var/lib/kubelet/pods/3e6689e8-ec4d-4b52-a443-e882316b71f8/volumes" Nov 03 22:32:22.731181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:22.731165 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd44272-3cab-46a1-92d0-e3a5849156d8" path="/var/lib/kubelet/pods/5fd44272-3cab-46a1-92d0-e3a5849156d8/volumes" Nov 03 22:32:27.791502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:27.791473 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 22:32:32.187611 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:32.187570 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:32:33.246838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.246803 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-e1805d"] Nov 03 22:32:33.248053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.248029 4625 cpu_manager.go:414] "containerMap: removing container" podUID="5fd44272-3cab-46a1-92d0-e3a5849156d8" containerName="payload-processing" Nov 03 22:32:33.248128 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.248054 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd44272-3cab-46a1-92d0-e3a5849156d8" containerName="payload-processing" Nov 03 22:32:33.248128 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.248070 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3e6689e8-ec4d-4b52-a443-e882316b71f8" containerName="payload-pre-processing" Nov 03 22:32:33.248128 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.248078 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6689e8-ec4d-4b52-a443-e882316b71f8" containerName="payload-pre-processing" Nov 03 22:32:33.248265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.248252 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd44272-3cab-46a1-92d0-e3a5849156d8" containerName="payload-processing" Nov 03 22:32:33.248300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.248269 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6689e8-ec4d-4b52-a443-e882316b71f8" containerName="payload-pre-processing" Nov 03 22:32:33.252946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.252925 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:33.268387 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.268359 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-e1805d"] Nov 03 22:32:33.312962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.312934 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-khmwt\"" Nov 03 22:32:33.371745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.371698 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbs4\" (UniqueName: \"kubernetes.io/projected/4f31c80e-a3c9-4667-9461-e82006aab821-kube-api-access-fkbs4\") pod \"test-curl-7269-e1805d\" (UID: \"4f31c80e-a3c9-4667-9461-e82006aab821\") " pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:33.474843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.474800 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbs4\" (UniqueName: \"kubernetes.io/projected/4f31c80e-a3c9-4667-9461-e82006aab821-kube-api-access-fkbs4\") pod \"test-curl-7269-e1805d\" (UID: \"4f31c80e-a3c9-4667-9461-e82006aab821\") " pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:33.497262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.497195 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbs4\" (UniqueName: \"kubernetes.io/projected/4f31c80e-a3c9-4667-9461-e82006aab821-kube-api-access-fkbs4\") pod \"test-curl-7269-e1805d\" (UID: \"4f31c80e-a3c9-4667-9461-e82006aab821\") " pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:33.572039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.571997 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:33.883338 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:33.883308 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-e1805d"] Nov 03 22:32:33.885263 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:32:33.885234 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f31c80e_a3c9_4667_9461_e82006aab821.slice/crio-7d6bd66bda9e172b70c28123296f5bb8b2edaf8ddd0bcbaa6bcfc5139232b927 WatchSource:0}: Error finding container 7d6bd66bda9e172b70c28123296f5bb8b2edaf8ddd0bcbaa6bcfc5139232b927: Status 404 returned error can't find the container with id 7d6bd66bda9e172b70c28123296f5bb8b2edaf8ddd0bcbaa6bcfc5139232b927 Nov 03 22:32:34.004927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:34.004887 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-e1805d" event={"ID":"4f31c80e-a3c9-4667-9461-e82006aab821","Type":"ContainerStarted","Data":"148fb02d97867280fac97ad2968b7b1aaa3a136b48b4699ebe8ae7203980cf01"} Nov 03 22:32:34.004927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:34.004932 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-e1805d" event={"ID":"4f31c80e-a3c9-4667-9461-e82006aab821","Type":"ContainerStarted","Data":"7d6bd66bda9e172b70c28123296f5bb8b2edaf8ddd0bcbaa6bcfc5139232b927"} Nov 03 22:32:34.044940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:34.044872 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-e1805d" podStartSLOduration=1.044857084 podStartE2EDuration="1.044857084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:34.042068479 +0000 UTC m=+1175.471404471" watchObservedRunningTime="2026-11-03 22:32:34.044857084 +0000 UTC m=+1175.474193077" Nov 03 22:32:35.006499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.006415 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-e1805d"] Nov 03 22:32:35.016040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.013338 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-e1805d"] Nov 03 22:32:35.158223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.158183 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-619395"] Nov 03 22:32:35.158963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.158946 4625 cpu_manager.go:414] "containerMap: removing container" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" containerName="test-curl-7269-e1805d" Nov 03 22:32:35.158963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.158963 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" containerName="test-curl-7269-e1805d" Nov 03 22:32:35.159127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.159114 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" containerName="test-curl-7269-e1805d" Nov 03 22:32:35.161876 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.161857 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:35.187490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.187462 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-619395"] Nov 03 22:32:35.218155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.218121 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kts2h\" (UniqueName: \"kubernetes.io/projected/f6ddba07-72a9-4b91-95cb-32018a064dbc-kube-api-access-kts2h\") pod \"test-curl-7269-619395\" (UID: \"f6ddba07-72a9-4b91-95cb-32018a064dbc\") " pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:35.319492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.319405 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kts2h\" (UniqueName: \"kubernetes.io/projected/f6ddba07-72a9-4b91-95cb-32018a064dbc-kube-api-access-kts2h\") pod \"test-curl-7269-619395\" (UID: \"f6ddba07-72a9-4b91-95cb-32018a064dbc\") " pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:35.342151 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.342115 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kts2h\" (UniqueName: \"kubernetes.io/projected/f6ddba07-72a9-4b91-95cb-32018a064dbc-kube-api-access-kts2h\") pod \"test-curl-7269-619395\" (UID: \"f6ddba07-72a9-4b91-95cb-32018a064dbc\") " pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:35.473874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.473838 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:35.789389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:35.789360 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-619395"] Nov 03 22:32:35.790424 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:32:35.790394 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ddba07_72a9_4b91_95cb_32018a064dbc.slice/crio-d1cd9c890daaf1348c391943792354484ced9e414f6f818f661cd5f2b361d328 WatchSource:0}: Error finding container d1cd9c890daaf1348c391943792354484ced9e414f6f818f661cd5f2b361d328: Status 404 returned error can't find the container with id d1cd9c890daaf1348c391943792354484ced9e414f6f818f661cd5f2b361d328 Nov 03 22:32:36.081788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:36.077183 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-e1805d" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" containerName="test-curl-7269-e1805d" containerID="cri-o://148fb02d97867280fac97ad2968b7b1aaa3a136b48b4699ebe8ae7203980cf01" gracePeriod=2 Nov 03 22:32:36.081788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:36.078492 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-619395" event={"ID":"f6ddba07-72a9-4b91-95cb-32018a064dbc","Type":"ContainerStarted","Data":"4e1a51a28a7d512505ac41103bc1cea0a64781f7b6dcc20d9880664e82e372be"} Nov 03 22:32:36.081788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:36.078545 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-619395" event={"ID":"f6ddba07-72a9-4b91-95cb-32018a064dbc","Type":"ContainerStarted","Data":"d1cd9c890daaf1348c391943792354484ced9e414f6f818f661cd5f2b361d328"} Nov 03 22:32:36.174517 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:36.173972 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-e1805d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:36.190081 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:36.190021 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-e1805d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:37.452655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.452583 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-619395" podStartSLOduration=2.452563694 podStartE2EDuration="2.452563694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:36.160317574 +0000 UTC m=+1177.589653544" watchObservedRunningTime="2026-11-03 22:32:37.452563694 +0000 UTC m=+1178.881899690" Nov 03 22:32:37.457792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.457377 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-619395"] Nov 03 22:32:37.483738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.480243 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-619395"] Nov 03 22:32:37.972880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.971817 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-0cfc4d"] Nov 03 22:32:37.990117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.973147 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f6ddba07-72a9-4b91-95cb-32018a064dbc" containerName="test-curl-7269-619395" Nov 03 22:32:37.990117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.973174 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ddba07-72a9-4b91-95cb-32018a064dbc" containerName="test-curl-7269-619395" Nov 03 22:32:37.990117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.973395 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ddba07-72a9-4b91-95cb-32018a064dbc" containerName="test-curl-7269-619395" Nov 03 22:32:37.990117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.983310 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-0cfc4d"] Nov 03 22:32:37.990117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:37.983422 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:38.099540 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:38.090184 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmrb\" (UniqueName: \"kubernetes.io/projected/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16-kube-api-access-frmrb\") pod \"test-curl-7269-0cfc4d\" (UID: \"fec4b92a-7e03-4ba1-9ecb-b49183b0ad16\") " pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:38.211760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:38.208136 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmrb\" (UniqueName: \"kubernetes.io/projected/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16-kube-api-access-frmrb\") pod \"test-curl-7269-0cfc4d\" (UID: \"fec4b92a-7e03-4ba1-9ecb-b49183b0ad16\") " pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:38.213801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:38.213760 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-619395" podUID="f6ddba07-72a9-4b91-95cb-32018a064dbc" containerName="test-curl-7269-619395" containerID="cri-o://4e1a51a28a7d512505ac41103bc1cea0a64781f7b6dcc20d9880664e82e372be" gracePeriod=2 Nov 03 22:32:38.229456 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:38.229333 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-619395\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="f6ddba07-72a9-4b91-95cb-32018a064dbc" pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:38.296759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:38.273553 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmrb\" (UniqueName: \"kubernetes.io/projected/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16-kube-api-access-frmrb\") pod \"test-curl-7269-0cfc4d\" (UID: \"fec4b92a-7e03-4ba1-9ecb-b49183b0ad16\") " pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:38.355739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:38.333098 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:38.637466 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:38.636841 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:32:38.680742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:38.669834 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Nov 03 22:32:39.000208 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:39.000161 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-619395\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="f6ddba07-72a9-4b91-95cb-32018a064dbc" pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:39.096861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:39.096810 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:32:39.334800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:39.334377 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:39.639928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:39.639736 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:39.640770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:39.640070 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:39.671328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:39.671298 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:32:40.101744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:40.101338 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:40.102753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:40.102107 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:32:40.355788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:40.352828 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:40.355788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:40.353494 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:32:40.642212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:40.642106 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:41.110291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:41.101824 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:41.648883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:41.369304 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:32:41.699740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:41.683271 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:41.933212 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:41.731996 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ddba07_72a9_4b91_95cb_32018a064dbc.slice/crio-conmon-4e1a51a28a7d512505ac41103bc1cea0a64781f7b6dcc20d9880664e82e372be.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:32:41.933212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:41.742874 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:41.943481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:41.941862 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:42.192837 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:42.129314 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:42.521889 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:42.503067 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:42.679960 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:42.667257 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-0cfc4d"] Nov 03 22:32:42.744867 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:42.744818 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:42.758512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:42.747399 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:43.065757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:43.042474 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:32:43.065757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:43.042542 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:32:43.562070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:43.561968 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:43.806869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:43.805257 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:32:43.806869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:43.805578 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:44.574875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.573565 4625 generic.go:356] "Generic (PLEG): container finished" podID="f6ddba07-72a9-4b91-95cb-32018a064dbc" containerID="4e1a51a28a7d512505ac41103bc1cea0a64781f7b6dcc20d9880664e82e372be" exitCode=137 Nov 03 22:32:44.662701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.660435 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qt78b" podUID="2d8e5e86-4926-41f1-ba99-b2da2fab4889" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 22:32:44.679236 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.679192 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:44.681406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.681365 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-0cfc4d" event={"ID":"fec4b92a-7e03-4ba1-9ecb-b49183b0ad16","Type":"ContainerStarted","Data":"6c5a813f49bedf273ce80c1b595e809ce3185ae516612ded4f5a21dc47df161f"} Nov 03 22:32:44.719781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.713108 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:32:44.737567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.729123 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:44.758547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.758505 4625 generic.go:356] "Generic (PLEG): container finished" podID="4f31c80e-a3c9-4667-9461-e82006aab821" containerID="148fb02d97867280fac97ad2968b7b1aaa3a136b48b4699ebe8ae7203980cf01" exitCode=137 Nov 03 22:32:44.792740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.775313 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ddba07-72a9-4b91-95cb-32018a064dbc-kube-api-access-kts2h\" (UniqueName: \"kubernetes.io/projected/f6ddba07-72a9-4b91-95cb-32018a064dbc-kube-api-access-kts2h\") pod \"f6ddba07-72a9-4b91-95cb-32018a064dbc\" (UID: \"f6ddba07-72a9-4b91-95cb-32018a064dbc\") " Nov 03 22:32:44.810702 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.810190 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:44.830841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.818937 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ddba07-72a9-4b91-95cb-32018a064dbc-kube-api-access-kts2h" pod "f6ddba07-72a9-4b91-95cb-32018a064dbc" (UID: "f6ddba07-72a9-4b91-95cb-32018a064dbc"). InnerVolumeSpecName "kube-api-access-kts2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:44.853840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.851828 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:44.885670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.882696 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:44.885670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.884123 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kts2h\" (UniqueName: \"kubernetes.io/projected/f6ddba07-72a9-4b91-95cb-32018a064dbc-kube-api-access-kts2h\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:44.885670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.884172 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:44.987749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:44.975090 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:45.027913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.020166 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ddba07-72a9-4b91-95cb-32018a064dbc" path="/var/lib/kubelet/pods/f6ddba07-72a9-4b91-95cb-32018a064dbc/volumes" Nov 03 22:32:45.032134 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.031505 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:32:45.821976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.821832 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:45.892530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.892493 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:45.927239 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:45.927193 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-e1805d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:45.933029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.932985 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-0cfc4d" event={"ID":"fec4b92a-7e03-4ba1-9ecb-b49183b0ad16","Type":"ContainerStarted","Data":"4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a"} Nov 03 22:32:45.965522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.963876 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f31c80e-a3c9-4667-9461-e82006aab821-kube-api-access-fkbs4\" (UniqueName: \"kubernetes.io/projected/4f31c80e-a3c9-4667-9461-e82006aab821-kube-api-access-fkbs4\") pod \"4f31c80e-a3c9-4667-9461-e82006aab821\" (UID: \"4f31c80e-a3c9-4667-9461-e82006aab821\") " Nov 03 22:32:45.966438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.966411 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:45.972659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.972630 4625 scope.go:122] "RemoveContainer" containerID="148fb02d97867280fac97ad2968b7b1aaa3a136b48b4699ebe8ae7203980cf01" Nov 03 22:32:45.984267 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:45.983533 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-619395" Nov 03 22:32:45.999621 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:45.999584 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-e1805d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:46.032459 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:46.032405 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-e1805d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:46.079277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:46.056242 4625 scope.go:122] "RemoveContainer" containerID="4e1a51a28a7d512505ac41103bc1cea0a64781f7b6dcc20d9880664e82e372be" Nov 03 22:32:46.079277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:46.076656 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:32:46.127869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:46.127658 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f31c80e-a3c9-4667-9461-e82006aab821-kube-api-access-fkbs4" pod "4f31c80e-a3c9-4667-9461-e82006aab821" (UID: "4f31c80e-a3c9-4667-9461-e82006aab821"). InnerVolumeSpecName "kube-api-access-fkbs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:46.141925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:46.138199 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-0cfc4d" podStartSLOduration=9.138173666 podStartE2EDuration="9.138173666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:45.987129175 +0000 UTC m=+1187.416465169" watchObservedRunningTime="2026-11-03 22:32:46.138173666 +0000 UTC m=+1187.567509660" Nov 03 22:32:46.165637 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:46.165600 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-e1805d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:46.193899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:46.181996 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkbs4\" (UniqueName: \"kubernetes.io/projected/4f31c80e-a3c9-4667-9461-e82006aab821-kube-api-access-fkbs4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:46.700742 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:46.690672 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-e1805d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" pod="opendatahub/test-curl-7269-e1805d" Nov 03 22:32:46.820074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:46.820036 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f31c80e-a3c9-4667-9461-e82006aab821" path="/var/lib/kubelet/pods/4f31c80e-a3c9-4667-9461-e82006aab821/volumes" Nov 03 22:32:48.237859 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.237801 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-0cfc4d"] Nov 03 22:32:48.322738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.316693 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-0cfc4d" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" containerName="test-curl-7269-0cfc4d" containerID="cri-o://4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a" gracePeriod=2 Nov 03 22:32:48.330152 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:48.330113 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-0cfc4d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:48.427882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.427825 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-0cfc4d"] Nov 03 22:32:48.523350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.521838 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-4d321d"] Nov 03 22:32:48.523350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.523150 4625 cpu_manager.go:414] "containerMap: removing container" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" containerName="test-curl-7269-0cfc4d" Nov 03 22:32:48.523350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.523176 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" containerName="test-curl-7269-0cfc4d" Nov 03 22:32:48.523652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.523392 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" containerName="test-curl-7269-0cfc4d" Nov 03 22:32:48.568753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.568140 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:48.589231 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:48.589183 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-0cfc4d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:48.717742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.652026 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-4d321d"] Nov 03 22:32:48.717742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.692826 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4hd\" (UniqueName: \"kubernetes.io/projected/3308b700-e465-48ef-8cf2-a4057b1021f9-kube-api-access-dx4hd\") pod \"test-curl-7269-4d321d\" (UID: \"3308b700-e465-48ef-8cf2-a4057b1021f9\") " pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:48.767521 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:48.766859 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-0cfc4d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:48.844803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.840816 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4hd\" (UniqueName: \"kubernetes.io/projected/3308b700-e465-48ef-8cf2-a4057b1021f9-kube-api-access-dx4hd\") pod \"test-curl-7269-4d321d\" (UID: \"3308b700-e465-48ef-8cf2-a4057b1021f9\") " pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:48.903565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:48.886195 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4hd\" (UniqueName: \"kubernetes.io/projected/3308b700-e465-48ef-8cf2-a4057b1021f9-kube-api-access-dx4hd\") pod \"test-curl-7269-4d321d\" (UID: \"3308b700-e465-48ef-8cf2-a4057b1021f9\") " pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:49.100162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:49.099774 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:49.616569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:49.616471 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-4d321d"] Nov 03 22:32:49.617043 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:32:49.616841 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3308b700_e465_48ef_8cf2_a4057b1021f9.slice/crio-8f83a8715c27e43075d1d406152c898b5412df039988ac8da5fed19e3cf71389 WatchSource:0}: Error finding container 8f83a8715c27e43075d1d406152c898b5412df039988ac8da5fed19e3cf71389: Status 404 returned error can't find the container with id 8f83a8715c27e43075d1d406152c898b5412df039988ac8da5fed19e3cf71389 Nov 03 22:32:50.436621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:50.436581 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-4d321d" event={"ID":"3308b700-e465-48ef-8cf2-a4057b1021f9","Type":"ContainerStarted","Data":"0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9"} Nov 03 22:32:50.436834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:50.436637 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-4d321d" event={"ID":"3308b700-e465-48ef-8cf2-a4057b1021f9","Type":"ContainerStarted","Data":"8f83a8715c27e43075d1d406152c898b5412df039988ac8da5fed19e3cf71389"} Nov 03 22:32:50.480962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:50.480887 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-4d321d" podStartSLOduration=2.48086734 podStartE2EDuration="2.48086734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:50.473416501 +0000 UTC m=+1191.902752494" watchObservedRunningTime="2026-11-03 22:32:50.48086734 +0000 UTC m=+1191.910203339" Nov 03 22:32:50.816765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:50.816730 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:50.855120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:50.855086 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16-kube-api-access-frmrb\" (UniqueName: \"kubernetes.io/projected/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16-kube-api-access-frmrb\") pod \"fec4b92a-7e03-4ba1-9ecb-b49183b0ad16\" (UID: \"fec4b92a-7e03-4ba1-9ecb-b49183b0ad16\") " Nov 03 22:32:50.857943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:50.857907 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16-kube-api-access-frmrb" pod "fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" (UID: "fec4b92a-7e03-4ba1-9ecb-b49183b0ad16"). InnerVolumeSpecName "kube-api-access-frmrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:50.957530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:50.957490 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frmrb\" (UniqueName: \"kubernetes.io/projected/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16-kube-api-access-frmrb\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:51.459077 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:51.459041 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-4d321d"] Nov 03 22:32:51.472529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:51.472484 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-4d321d"] Nov 03 22:32:51.485891 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:51.485851 4625 generic.go:356] "Generic (PLEG): container finished" podID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" containerID="4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a" exitCode=137 Nov 03 22:32:51.487113 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:51.487067 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:51.491163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:51.490929 4625 scope.go:122] "RemoveContainer" containerID="4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a" Nov 03 22:32:51.516432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:51.513753 4625 scope.go:122] "RemoveContainer" containerID="4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a" Nov 03 22:32:51.518113 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:51.518076 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a\": container with ID starting with 4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a not found: ID does not exist" containerID="4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a" Nov 03 22:32:51.518257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:51.518125 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a"} err="failed to get container status \"4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a\": rpc error: code = NotFound desc = could not find container \"4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a\": container with ID starting with 4558621a968ab403813b3dc51ba9c242da80bd568f58cb0f16ca6d3c327a104a not found: ID does not exist" Nov 03 22:32:51.549099 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:51.547425 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-0cfc4d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:52.504555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:52.504479 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-4d321d" podUID="3308b700-e465-48ef-8cf2-a4057b1021f9" containerName="test-curl-7269-4d321d" containerID="cri-o://0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9" gracePeriod=2 Nov 03 22:32:52.518004 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:52.517955 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-0cfc4d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" pod="opendatahub/test-curl-7269-0cfc4d" Nov 03 22:32:52.530363 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:52.530324 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-4d321d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="3308b700-e465-48ef-8cf2-a4057b1021f9" pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:52.971741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:52.968319 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec4b92a-7e03-4ba1-9ecb-b49183b0ad16" path="/var/lib/kubelet/pods/fec4b92a-7e03-4ba1-9ecb-b49183b0ad16/volumes" Nov 03 22:32:53.434746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.433054 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j"] Nov 03 22:32:53.434746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.434316 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3308b700-e465-48ef-8cf2-a4057b1021f9" containerName="test-curl-7269-4d321d" Nov 03 22:32:53.434746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.434338 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3308b700-e465-48ef-8cf2-a4057b1021f9" containerName="test-curl-7269-4d321d" Nov 03 22:32:53.434746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.434585 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3308b700-e465-48ef-8cf2-a4057b1021f9" containerName="test-curl-7269-4d321d" Nov 03 22:32:53.473738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.465113 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.476984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.476904 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j"] Nov 03 22:32:53.484763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.484500 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-2a2eca-openshift-default-dockercfg-t2dd6\"" Nov 03 22:32:53.495520 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495485 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-credential-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.495700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495582 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb83ca-1c71-40d1-a181-88d931f3538d-istiod-ca-cert\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.495700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495628 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-data\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.495700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495681 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-certs\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.495939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495752 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-envoy\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.495939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495800 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-podinfo\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.495939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495893 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg22d\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-kube-api-access-pg22d\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.496095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.495945 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-token\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.496095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.496024 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.596939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.596901 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb83ca-1c71-40d1-a181-88d931f3538d-istiod-ca-cert\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.596953 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-data\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.596993 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-certs\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597018 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-envoy\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597051 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-podinfo\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597108 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg22d\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-kube-api-access-pg22d\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597139 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-token\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597190 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.597445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597248 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-credential-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.598291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597647 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-credential-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.598291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597855 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb83ca-1c71-40d1-a181-88d931f3538d-istiod-ca-cert\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.598291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.597982 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.598291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.598057 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-certs\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.598291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.598125 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-data\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.600104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.600083 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-podinfo\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.600521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.600505 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-envoy\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.675769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.675590 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-token\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.692272 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.691505 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg22d\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-kube-api-access-pg22d\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:53.828409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:53.828075 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:54.436026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.435982 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf"] Nov 03 22:32:54.471971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.471932 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf"] Nov 03 22:32:54.472230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.472053 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.600650 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694ccfdd-7723-4d33-b5c8-98585ffcb013-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.600830 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.600918 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.601091 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.601219 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.601349 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.601436 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.601464 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.629253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.601498 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnpw\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-kube-api-access-4dnpw\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.710924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.710866 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.711227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.711206 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.711449 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.711431 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.711641 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.711625 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.711858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.711837 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.712024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.712010 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.712143 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.712116 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.712287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.712272 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnpw\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-kube-api-access-4dnpw\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.718736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.712536 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694ccfdd-7723-4d33-b5c8-98585ffcb013-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.755133 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.755097 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694ccfdd-7723-4d33-b5c8-98585ffcb013-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.862742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.829440 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.862742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.829858 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.862742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.830108 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.891511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.891428 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j"] Nov 03 22:32:54.975854 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.895551 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.975854 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.920627 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:54.975854 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.945030 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:54.975854 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.945128 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:54.975854 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:54.945173 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:55.043192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.043113 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:55.115962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.115919 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnpw\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-kube-api-access-4dnpw\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:55.121924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.121637 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:55.177006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.176764 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:55.397747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.397506 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:55.440620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.440529 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b"] Nov 03 22:32:55.464576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.464529 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.497667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.497633 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b"] Nov 03 22:32:55.516697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.516077 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3308b700-e465-48ef-8cf2-a4057b1021f9-kube-api-access-dx4hd\" (UniqueName: \"kubernetes.io/projected/3308b700-e465-48ef-8cf2-a4057b1021f9-kube-api-access-dx4hd\") pod \"3308b700-e465-48ef-8cf2-a4057b1021f9\" (UID: \"3308b700-e465-48ef-8cf2-a4057b1021f9\") " Nov 03 22:32:55.521261 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.519472 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3308b700-e465-48ef-8cf2-a4057b1021f9-kube-api-access-dx4hd" pod "3308b700-e465-48ef-8cf2-a4057b1021f9" (UID: "3308b700-e465-48ef-8cf2-a4057b1021f9"). InnerVolumeSpecName "kube-api-access-dx4hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:55.522507 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.522457 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4hd\" (UniqueName: \"kubernetes.io/projected/3308b700-e465-48ef-8cf2-a4057b1021f9-kube-api-access-dx4hd\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:32:55.623771 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.623730 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.623866 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.623836 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.624009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.623983 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.624103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.624029 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4vsd\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-kube-api-access-z4vsd\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.624168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.624113 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.624221 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.624162 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3babc-c24a-417f-9dd9-cc332cf6d463-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.624326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.624302 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.624391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.624359 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.624518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.624495 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.671311 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:32:55.671270 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694ccfdd_7723_4d33_b5c8_98585ffcb013.slice/crio-49798e2634606ef69862b60e8ea6bf79cfd5f6fa42959926035a16551d892a7b WatchSource:0}: Error finding container 49798e2634606ef69862b60e8ea6bf79cfd5f6fa42959926035a16551d892a7b: Status 404 returned error can't find the container with id 49798e2634606ef69862b60e8ea6bf79cfd5f6fa42959926035a16551d892a7b Nov 03 22:32:55.672041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.672018 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf"] Nov 03 22:32:55.674790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.674753 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:55.674911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.674826 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:55.674911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.674871 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:55.726230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726192 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726293 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726409 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726503 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726530 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4vsd\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-kube-api-access-z4vsd\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726584 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726609 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3babc-c24a-417f-9dd9-cc332cf6d463-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726706 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.726997 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.726816 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.727206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.727169 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.727291 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.727247 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.727448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.727425 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.728016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.727993 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.728569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.728542 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3babc-c24a-417f-9dd9-cc332cf6d463-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.729924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.729900 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.730916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.730894 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.758928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.758893 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4vsd\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-kube-api-access-z4vsd\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.777361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.777325 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:55.831858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:55.831823 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:56.058361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.058006 4625 generic.go:356] "Generic (PLEG): container finished" podID="3308b700-e465-48ef-8cf2-a4057b1021f9" containerID="0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9" exitCode=137 Nov 03 22:32:56.058361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.058159 4625 scope.go:122] "RemoveContainer" containerID="0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9" Nov 03 22:32:56.060256 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.058706 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:56.066817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.066666 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" event={"ID":"7ceb83ca-1c71-40d1-a181-88d931f3538d","Type":"ContainerStarted","Data":"f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094"} Nov 03 22:32:56.068041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.066981 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" event={"ID":"7ceb83ca-1c71-40d1-a181-88d931f3538d","Type":"ContainerStarted","Data":"26648bd369e3a0f4549eb0bf9760040ccadfc7adba569b5a4550adb6eaa39f05"} Nov 03 22:32:56.074296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.073301 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" event={"ID":"694ccfdd-7723-4d33-b5c8-98585ffcb013","Type":"ContainerStarted","Data":"d2dbadc7964719a21c09e3bcfc6fe2c033663ac7436116997406ea3662d5b9c6"} Nov 03 22:32:56.074296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.073354 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" event={"ID":"694ccfdd-7723-4d33-b5c8-98585ffcb013","Type":"ContainerStarted","Data":"49798e2634606ef69862b60e8ea6bf79cfd5f6fa42959926035a16551d892a7b"} Nov 03 22:32:56.137761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.136953 4625 scope.go:122] "RemoveContainer" containerID="0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9" Nov 03 22:32:56.140189 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:56.140151 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9\": container with ID starting with 0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9 not found: ID does not exist" containerID="0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9" Nov 03 22:32:56.140340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.140202 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9"} err="failed to get container status \"0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9\": rpc error: code = NotFound desc = could not find container \"0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9\": container with ID starting with 0ef8ab32ea83f89f862f28672e7e7903786cd4fdd084e44f4ebb948e02b36ab9 not found: ID does not exist" Nov 03 22:32:56.144109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.143857 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podStartSLOduration=3.143840244 podStartE2EDuration="3.143840244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:56.139109077 +0000 UTC m=+1197.568445078" watchObservedRunningTime="2026-11-03 22:32:56.143840244 +0000 UTC m=+1197.573176237" Nov 03 22:32:56.177818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.177780 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:32:56.179902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.179871 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:32:56.180042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.179934 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:32:56.190374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.190307 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podStartSLOduration=2.190286319 podStartE2EDuration="2.190286319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:56.182111263 +0000 UTC m=+1197.611447271" watchObservedRunningTime="2026-11-03 22:32:56.190286319 +0000 UTC m=+1197.619622313" Nov 03 22:32:56.199073 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:32:56.199020 4625 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-4d321d\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="3308b700-e465-48ef-8cf2-a4057b1021f9" pod="opendatahub/test-curl-7269-4d321d" Nov 03 22:32:56.267500 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.267007 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b"] Nov 03 22:32:56.269186 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:32:56.269150 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb3babc_c24a_417f_9dd9_cc332cf6d463.slice/crio-ada7ff7add01a1da62a08a7a458cf5fc740d3677ed7c8663588af3992f44f953 WatchSource:0}: Error finding container ada7ff7add01a1da62a08a7a458cf5fc740d3677ed7c8663588af3992f44f953: Status 404 returned error can't find the container with id ada7ff7add01a1da62a08a7a458cf5fc740d3677ed7c8663588af3992f44f953 Nov 03 22:32:56.273792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.273749 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:56.273924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.273854 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:56.273998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.273911 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:56.778144 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.771980 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3308b700-e465-48ef-8cf2-a4057b1021f9" path="/var/lib/kubelet/pods/3308b700-e465-48ef-8cf2-a4057b1021f9/volumes" Nov 03 22:32:56.828690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.828650 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:56.841831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:56.839059 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:57.094048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.093952 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" event={"ID":"deb3babc-c24a-417f-9dd9-cc332cf6d463","Type":"ContainerStarted","Data":"e32c5bbe286e89493e9514096522f046ce91736908dcfa8ef80c36ba391540a4"} Nov 03 22:32:57.094048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.094035 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" event={"ID":"deb3babc-c24a-417f-9dd9-cc332cf6d463","Type":"ContainerStarted","Data":"ada7ff7add01a1da62a08a7a458cf5fc740d3677ed7c8663588af3992f44f953"} Nov 03 22:32:57.100776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.100708 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:57.101819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.101791 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:32:57.180515 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.180241 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:32:57.180515 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.180294 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:32:57.188126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.188030 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podStartSLOduration=2.188010611 podStartE2EDuration="2.188010611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:57.182058831 +0000 UTC m=+1198.611394823" watchObservedRunningTime="2026-11-03 22:32:57.188010611 +0000 UTC m=+1198.617346605" Nov 03 22:32:57.843616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.843532 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:32:57.852785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.846830 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:32:57.852785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:57.846894 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:32:58.002029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.001941 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:32:58.002264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.002238 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:32:58.111970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.111936 4625 generic.go:356] "Generic (PLEG): container finished" podID="652d1897-361b-493e-a6ce-081d556893a7" containerID="450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8" exitCode=0 Nov 03 22:32:58.112139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.112122 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" event={"ID":"652d1897-361b-493e-a6ce-081d556893a7","Type":"ContainerDied","Data":"450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8"} Nov 03 22:32:58.112608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.112591 4625 scope.go:122] "RemoveContainer" containerID="450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8" Nov 03 22:32:58.115825 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.115789 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:58.115928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.115862 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:58.115928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.115908 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:58.116086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.116064 4625 generic.go:356] "Generic (PLEG): container finished" podID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerID="9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc" exitCode=0 Nov 03 22:32:58.116288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.116271 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" event={"ID":"f9903686-a262-41cd-b4fd-2bf5936cddca","Type":"ContainerDied","Data":"9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc"} Nov 03 22:32:58.117459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.117442 4625 scope.go:122] "RemoveContainer" containerID="9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc" Nov 03 22:32:58.127243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.127208 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:58.127361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.127297 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:58.127432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.127360 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:32:58.211765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.206687 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:32:58.211765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.206756 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:32:58.835898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.835861 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:32:58.836303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:58.836257 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:32:59.203742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.183415 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:32:59.224743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.207286 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:32:59.319777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.319504 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" event={"ID":"652d1897-361b-493e-a6ce-081d556893a7","Type":"ContainerStarted","Data":"03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125"} Nov 03 22:32:59.324411 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.324371 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" event={"ID":"f9903686-a262-41cd-b4fd-2bf5936cddca","Type":"ContainerStarted","Data":"69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2"} Nov 03 22:32:59.528847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.503967 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:32:59.528847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.504077 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:32:59.528847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.504115 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:32:59.854748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.844853 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:32:59.854748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:32:59.844919 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:33:00.177254 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.177161 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:33:00.177922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.177516 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:33:00.208010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.207975 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:33:00.208674 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.208650 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:00.208867 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.208842 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:00.490141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.490039 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:33:00.490482 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.490442 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:33:00.564271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.564229 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:33:01.021803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.983923 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:33:01.021803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:00.984499 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:33:01.264745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.264395 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:33:01.264745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.264481 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:33:01.311792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.309682 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:01.311792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.309762 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:01.449790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.385398 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:01.894759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.890413 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:33:01.894759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.890472 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:33:01.997638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.997577 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:01.999464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:01.999296 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Nov 03 22:33:02.087902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.086815 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:02.359744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.345030 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.359744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.345107 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:02.372621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.372073 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.372621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.372152 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:33:02.372621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.372241 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.372621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.372265 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:33:02.497091 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.484556 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.497091 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.484622 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:33:02.552583 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.552553 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:33:02.702308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.702225 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:33:02.797764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.797707 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:02.837565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.836794 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.837565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:02.836852 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:33:02.840213 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:33:02.840183 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:33:03.196112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.196065 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:03.233899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.211519 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:33:03.233899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.213143 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:33:03.233899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.213195 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:33:03.233899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.213561 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:03.233899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.213593 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:03.408914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.393345 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:33:03.523248 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.509541 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:33:03.523248 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.514954 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:33:03.847805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.831822 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:33:03.928778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.857953 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:33:03.928778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.888959 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:33:03.928778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.889036 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:33:03.990933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:03.981101 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:04.213272 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.213230 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:04.215648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.215018 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:04.215648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.215069 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:04.315287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.315245 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:33:04.315557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.315525 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 22:33:04.485167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.485064 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:04.509555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.488349 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:33:04.571751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.569937 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:33:04.571751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.569997 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:33:04.933838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:04.932169 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:05.010652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.010607 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:33:05.010955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.010926 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 22:33:05.059742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.052020 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 03 22:33:05.205524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.205106 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:05.205524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.205178 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:05.213700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.213658 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:05.236362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.236325 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:33:05.265707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.238702 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:33:05.270027 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.269997 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:33:05.406595 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.406556 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:33:05.421747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.408955 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:33:05.493794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.487855 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:05.493794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.488829 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:33:05.493794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.488878 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:33:05.816497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.816407 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:33:05.873994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.872230 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:33:05.873994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.873678 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:33:05.900743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:05.878584 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 22:33:06.205648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:06.205175 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:06.205648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:06.205238 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:06.499797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:06.499063 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:33:06.499797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:06.499126 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 22:33:07.232424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:07.231885 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 22:33:07.232424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:07.231935 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 22:33:07.612277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:07.612238 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:33:07.613298 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:07.613274 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:33:07.614082 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:07.614053 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:33:08.245314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:08.245245 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:33:08.247623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:08.247593 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 22:33:11.386532 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.386457 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj"] Nov 03 22:33:11.391372 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.391351 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.416372 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.415463 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj"] Nov 03 22:33:11.418146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.416126 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-23d281-serving-cert\"" Nov 03 22:33:11.499200 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.499166 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.499200 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.499199 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c53996e-be1a-4df1-a7c7-4bd4daecda71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.499414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.499307 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzdw\" (UniqueName: \"kubernetes.io/projected/1c53996e-be1a-4df1-a7c7-4bd4daecda71-kube-api-access-2hzdw\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.499523 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.499500 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.601862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.601826 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.602054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.601869 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c53996e-be1a-4df1-a7c7-4bd4daecda71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.602054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.601943 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzdw\" (UniqueName: \"kubernetes.io/projected/1c53996e-be1a-4df1-a7c7-4bd4daecda71-kube-api-access-2hzdw\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.602272 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.602100 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.602692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.602664 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c53996e-be1a-4df1-a7c7-4bd4daecda71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.604418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.604394 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.604619 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.604601 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.624164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.624139 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzdw\" (UniqueName: \"kubernetes.io/projected/1c53996e-be1a-4df1-a7c7-4bd4daecda71-kube-api-access-2hzdw\") pod \"maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:11.720864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:11.714385 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:12.268735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:12.263780 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj"] Nov 03 22:33:12.381783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:12.381747 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" event={"ID":"1c53996e-be1a-4df1-a7c7-4bd4daecda71","Type":"ContainerStarted","Data":"a116922ae616f2034f36a0ac04e8e3d33c90af410f30afc602c110d46a561a0b"} Nov 03 22:33:13.409976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:13.409933 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" event={"ID":"1c53996e-be1a-4df1-a7c7-4bd4daecda71","Type":"ContainerStarted","Data":"12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add"} Nov 03 22:33:13.410699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:13.410653 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:13.456616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:13.456558 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" podStartSLOduration=1.869814486 podStartE2EDuration="2.456543804s" totalImagesPullingTime="586.729318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:13.451246367 +0000 UTC m=+1214.880582357" watchObservedRunningTime="2026-11-03 22:33:13.456543804 +0000 UTC m=+1214.885880044" Nov 03 22:33:14.750804 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.750767 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs"] Nov 03 22:33:14.755498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.755460 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.781133 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.777693 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs"] Nov 03 22:33:14.781133 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.778382 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8b1efeb6-serving-cert\"" Nov 03 22:33:14.852077 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.852042 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-maas-api-tls\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.852302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.852275 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2b9\" (UniqueName: \"kubernetes.io/projected/0471975b-e9e0-41bc-af7a-b85fa6a87da2-kube-api-access-dm2b9\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.852456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.852436 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0471975b-e9e0-41bc-af7a-b85fa6a87da2-openshift-service-ca\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.852510 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.852489 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-metrics-certs\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.976756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.972876 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2b9\" (UniqueName: \"kubernetes.io/projected/0471975b-e9e0-41bc-af7a-b85fa6a87da2-kube-api-access-dm2b9\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.976756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.973054 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0471975b-e9e0-41bc-af7a-b85fa6a87da2-openshift-service-ca\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.976756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.973142 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-metrics-certs\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.976756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.973358 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-maas-api-tls\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:14.976756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:14.976664 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0471975b-e9e0-41bc-af7a-b85fa6a87da2-openshift-service-ca\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:15.008151 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:15.008062 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-metrics-certs\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:15.013847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:15.011433 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-maas-api-tls\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:15.017974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:15.017942 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2b9\" (UniqueName: \"kubernetes.io/projected/0471975b-e9e0-41bc-af7a-b85fa6a87da2-kube-api-access-dm2b9\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:15.080225 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:15.080193 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:15.549299 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:33:15.549263 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0471975b_e9e0_41bc_af7a_b85fa6a87da2.slice/crio-e2ab5f14010f08e9847cab81f259a30850b0b5ce050ba015bfecd2944713332e WatchSource:0}: Error finding container e2ab5f14010f08e9847cab81f259a30850b0b5ce050ba015bfecd2944713332e: Status 404 returned error can't find the container with id e2ab5f14010f08e9847cab81f259a30850b0b5ce050ba015bfecd2944713332e Nov 03 22:33:15.552656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:15.552626 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs"] Nov 03 22:33:16.564788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:16.564749 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" event={"ID":"0471975b-e9e0-41bc-af7a-b85fa6a87da2","Type":"ContainerStarted","Data":"9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b"} Nov 03 22:33:16.565289 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:16.564809 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" event={"ID":"0471975b-e9e0-41bc-af7a-b85fa6a87da2","Type":"ContainerStarted","Data":"e2ab5f14010f08e9847cab81f259a30850b0b5ce050ba015bfecd2944713332e"} Nov 03 22:33:16.565289 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:16.564883 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:16.608635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:16.608587 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" podStartSLOduration=2.27195099 podStartE2EDuration="2.608572302s" totalImagesPullingTime="336.621312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:16.607308898 +0000 UTC m=+1218.036644892" watchObservedRunningTime="2026-11-03 22:33:16.608572302 +0000 UTC m=+1218.037908326" Nov 03 22:33:19.545481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:19.545449 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 22:33:22.592655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:22.592624 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:33:23.458747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.450898 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5"] Nov 03 22:33:23.467751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.465601 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.483741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.480692 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-2a2eca-serving-cert\"" Nov 03 22:33:23.500755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.497705 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5"] Nov 03 22:33:23.570644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.570605 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.570951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.570919 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3123211a-c11f-4479-be91-2bf9edcb267e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.571381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.571363 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.571521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.571507 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjfc\" (UniqueName: \"kubernetes.io/projected/3123211a-c11f-4479-be91-2bf9edcb267e-kube-api-access-msjfc\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.743799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.690710 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.744804 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.744773 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3123211a-c11f-4479-be91-2bf9edcb267e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.745332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.745302 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.748739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.745497 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjfc\" (UniqueName: \"kubernetes.io/projected/3123211a-c11f-4479-be91-2bf9edcb267e-kube-api-access-msjfc\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.752743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:23.747630 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3123211a-c11f-4479-be91-2bf9edcb267e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:23.763739 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:33:23.747761 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-serving-cert: secret "maas-api-e2e-shared-a-w6-2a2eca-serving-cert" not found Nov 03 22:33:23.809760 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:33:23.764096 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls podName:3123211a-c11f-4479-be91-2bf9edcb267e nodeName:}" failed. No retries permitted until 2026-11-03 22:33:24.264066018 +0000 UTC m=+1225.693402003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls") pod "maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" (UID: "3123211a-c11f-4479-be91-2bf9edcb267e") : secret "maas-api-e2e-shared-a-w6-2a2eca-serving-cert" not found Nov 03 22:33:24.075001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.074924 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:24.107800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.107086 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjfc\" (UniqueName: \"kubernetes.io/projected/3123211a-c11f-4479-be91-2bf9edcb267e-kube-api-access-msjfc\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:24.219739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.218845 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:24.363207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.363162 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:24.421732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.421677 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:24.709795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.700031 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:24.768361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.768310 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:24.857908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:24.856994 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:25.037811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:25.037702 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:33:25.038058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:25.038026 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:26.231355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:26.231314 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5"] Nov 03 22:33:27.290836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:27.290743 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" event={"ID":"3123211a-c11f-4479-be91-2bf9edcb267e","Type":"ContainerStarted","Data":"9e75bba24adad5f148842f8ed0cea08656e2374a9387393d1aca01367bfe905f"} Nov 03 22:33:27.290836 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:27.290808 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" event={"ID":"3123211a-c11f-4479-be91-2bf9edcb267e","Type":"ContainerStarted","Data":"bcf6834d99a89353f70c343445824d46c8366a35f1f5c1af6e09c3100a2412f9"} Nov 03 22:33:27.291373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:27.290870 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:27.335112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:27.335059 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" podStartSLOduration=3.929341846 podStartE2EDuration="4.335043548s" totalImagesPullingTime="405.701702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:27.330603494 +0000 UTC m=+1228.759939488" watchObservedRunningTime="2026-11-03 22:33:27.335043548 +0000 UTC m=+1228.764379541" Nov 03 22:33:32.189465 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:33:32.189428 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:33:33.304550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:33.304520 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 22:33:41.463058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.463018 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6"] Nov 03 22:33:41.467832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.467800 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.512283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.512253 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6"] Nov 03 22:33:41.513385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.513356 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d8a3d1-openshift-default-dockercfg-c87pb\"" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.637614 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.637671 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-podinfo\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.637751 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-envoy\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.637790 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-certs\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.637950 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-credential-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.637991 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dk59\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-kube-api-access-7dk59\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.638045 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-data\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.638121 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-token\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.666764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.638167 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ab7891-a909-4113-8881-0dd30efd2d0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.741066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.740961 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-data\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.741381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.741356 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-token\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.741527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.741511 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ab7891-a909-4113-8881-0dd30efd2d0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.745740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.741813 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.745982 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.745955 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-podinfo\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.746154 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.746138 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-envoy\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.749734 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.746263 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-certs\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.769739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.750138 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-credential-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.770029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.770004 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dk59\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-kube-api-access-7dk59\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.770337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.747829 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ab7891-a909-4113-8881-0dd30efd2d0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.770457 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.759125 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.770542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.759706 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-credential-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.770632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.760382 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-certs\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.770731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.769277 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-podinfo\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.770843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.749382 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-data\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.771646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.771574 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-envoy\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.793447 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.793370 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-token\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:41.856786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:41.845657 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dk59\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-kube-api-access-7dk59\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:42.115971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:42.115906 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm"] Nov 03 22:33:42.131693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:42.131386 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:42.283153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:42.282985 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831ef85e-dhcqm"] Nov 03 22:33:42.780458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:42.780216 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c19a6f7-87bc-45e7-9e86-62b09ab69bee" path="/var/lib/kubelet/pods/1c19a6f7-87bc-45e7-9e86-62b09ab69bee/volumes" Nov 03 22:33:43.252431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:43.252394 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6"] Nov 03 22:33:43.252863 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:33:43.252841 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ab7891_a909_4113_8881_0dd30efd2d0c.slice/crio-739165cf54c222547de630f514f0f8204fa7cef9fe765f38935dcfd0c361056b WatchSource:0}: Error finding container 739165cf54c222547de630f514f0f8204fa7cef9fe765f38935dcfd0c361056b: Status 404 returned error can't find the container with id 739165cf54c222547de630f514f0f8204fa7cef9fe765f38935dcfd0c361056b Nov 03 22:33:43.256182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:43.256147 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:33:43.256293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:43.256211 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:33:43.256293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:43.256238 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:33:43.854436 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:43.854397 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" event={"ID":"b3ab7891-a909-4113-8881-0dd30efd2d0c","Type":"ContainerStarted","Data":"e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e"} Nov 03 22:33:43.854992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:43.854464 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" event={"ID":"b3ab7891-a909-4113-8881-0dd30efd2d0c","Type":"ContainerStarted","Data":"739165cf54c222547de630f514f0f8204fa7cef9fe765f38935dcfd0c361056b"} Nov 03 22:33:43.912794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:43.912732 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podStartSLOduration=2.9126989610000003 podStartE2EDuration="2.912698961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:43.908303006 +0000 UTC m=+1245.337638998" watchObservedRunningTime="2026-11-03 22:33:43.912698961 +0000 UTC m=+1245.342034955" Nov 03 22:33:44.203327 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:44.203233 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:44.219406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:44.219048 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:44.864073 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:44.864025 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:33:44.865052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:33:44.865031 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:34:00.447492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:00.447451 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2"] Nov 03 22:34:00.447996 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:00.447673 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" containerID="cri-o://8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd" gracePeriod=30 Nov 03 22:34:00.459251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:00.459219 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m"] Nov 03 22:34:00.459498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:00.459454 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" containerID="cri-o://cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4" gracePeriod=30 Nov 03 22:34:03.005515 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:03.005471 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:34:03.237891 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:03.237835 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:34:03.298792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:03.288870 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:34:03.298792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:03.288972 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:34:05.360133 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:05.360092 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:34:05.360858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:05.360823 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:34:05.826282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:05.826249 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:34:05.911773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:05.911674 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:34:06.014770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.014731 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b397c3-cce1-40d0-bd67-46c2b709bb99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b397c3-cce1-40d0-bd67-46c2b709bb99-istiod-ca-cert\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.014988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.014792 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-envoy\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.014988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.014846 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-socket\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.015239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015212 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b397c3-cce1-40d0-bd67-46c2b709bb99-istiod-ca-cert" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:06.015330 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015285 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-envoy\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.015394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015333 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-certs\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.015455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015400 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-certs\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.015514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015464 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-podinfo\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.015569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015541 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-socket" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.015624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015586 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-data\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.015684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015621 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-socket\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.015801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015688 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istiod-ca-cert\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.015801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015744 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-certs" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.015801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015747 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-token\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-token\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.015969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015821 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-token\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.015969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015859 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-credential-socket\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.015969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015895 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-kube-api-access-z2c26\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-kube-api-access-z2c26\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.015969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.015943 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-kube-api-access-vdfjj\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-kube-api-access-vdfjj\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.016179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.016062 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-credential-socket\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.016179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.016097 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-podinfo\") pod \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\" (UID: \"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a\") " Nov 03 22:34:06.016383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.016353 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-data\") pod \"86b397c3-cce1-40d0-bd67-46c2b709bb99\" (UID: \"86b397c3-cce1-40d0-bd67-46c2b709bb99\") " Nov 03 22:34:06.017262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017107 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-credential-socket" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.017363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017292 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-data" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.017363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017298 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-credential-socket" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.017975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017708 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-data" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.017975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017786 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-certs" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.017975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017925 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b397c3-cce1-40d0-bd67-46c2b709bb99-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.017975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017943 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.017975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017955 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.017975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017970 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.018401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.017984 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.018401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.018026 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.018401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.018039 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.018401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.018319 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istiod-ca-cert" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:06.018401 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.018390 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-socket" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.022351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.022317 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-podinfo" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:34:06.022938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.022765 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-podinfo" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:34:06.022938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.022862 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-kube-api-access-z2c26" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "kube-api-access-z2c26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:06.022938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.022874 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-kube-api-access-vdfjj" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "kube-api-access-vdfjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:06.022938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.022900 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-token" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:06.022938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.022900 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-envoy" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.023178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.022944 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-envoy" pod "86b397c3-cce1-40d0-bd67-46c2b709bb99" (UID: "86b397c3-cce1-40d0-bd67-46c2b709bb99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:06.024446 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.024409 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-token" pod "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" (UID: "8bfe3cdf-db10-474c-ab73-40bba8ab6b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:06.107218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.107168 4625 generic.go:356] "Generic (PLEG): container finished" podID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerID="cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4" exitCode=0 Nov 03 22:34:06.107218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.107212 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" Nov 03 22:34:06.107502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.107206 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" event={"ID":"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a","Type":"ContainerDied","Data":"cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4"} Nov 03 22:34:06.107502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.107336 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m" event={"ID":"8bfe3cdf-db10-474c-ab73-40bba8ab6b3a","Type":"ContainerDied","Data":"781b4f4a4a6c3129b934bbaa48e2a2e2c5e8f735342cff13b4daf36af13f06c2"} Nov 03 22:34:06.107502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.107360 4625 scope.go:122] "RemoveContainer" containerID="cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4" Nov 03 22:34:06.109563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.109538 4625 generic.go:356] "Generic (PLEG): container finished" podID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerID="8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd" exitCode=0 Nov 03 22:34:06.109693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.109642 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" Nov 03 22:34:06.109907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.109884 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" event={"ID":"86b397c3-cce1-40d0-bd67-46c2b709bb99","Type":"ContainerDied","Data":"8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd"} Nov 03 22:34:06.109971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.109933 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2" event={"ID":"86b397c3-cce1-40d0-bd67-46c2b709bb99","Type":"ContainerDied","Data":"b4fd955e72e4152caf8d18980497dcdcd6c196bdfa327182367d2fa6adb0c7d5"} Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119858 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119888 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119901 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119915 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119927 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119940 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119955 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119971 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119985 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2c26\" (UniqueName: \"kubernetes.io/projected/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-kube-api-access-z2c26\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.119999 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfjj\" (UniqueName: \"kubernetes.io/projected/86b397c3-cce1-40d0-bd67-46c2b709bb99-kube-api-access-vdfjj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.123282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.120012 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:06.127902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.127878 4625 scope.go:122] "RemoveContainer" containerID="cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4" Nov 03 22:34:06.128255 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:06.128231 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4\": container with ID starting with cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4 not found: ID does not exist" containerID="cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4" Nov 03 22:34:06.128328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.128266 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4"} err="failed to get container status \"cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4\": rpc error: code = NotFound desc = could not find container \"cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4\": container with ID starting with cc2ce935886ec1e675ace3dede5a92b522300439dcc8be72c757c544138829a4 not found: ID does not exist" Nov 03 22:34:06.128328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.128283 4625 scope.go:122] "RemoveContainer" containerID="8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd" Nov 03 22:34:06.177860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.177833 4625 scope.go:122] "RemoveContainer" containerID="8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd" Nov 03 22:34:06.178223 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:06.178191 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd\": container with ID starting with 8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd not found: ID does not exist" containerID="8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd" Nov 03 22:34:06.178313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.178232 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd"} err="failed to get container status \"8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd\": rpc error: code = NotFound desc = could not find container \"8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd\": container with ID starting with 8d297423ead8265bacd6a8189e059cca335bcea3c69c236a7eab8a02401a53cd not found: ID does not exist" Nov 03 22:34:06.179145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.179125 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2"] Nov 03 22:34:06.187519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.187490 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-g4fx2"] Nov 03 22:34:06.209922 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.209883 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m"] Nov 03 22:34:06.215181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.215143 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-vlq4m"] Nov 03 22:34:06.858673 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.856159 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" path="/var/lib/kubelet/pods/86b397c3-cce1-40d0-bd67-46c2b709bb99/volumes" Nov 03 22:34:06.858673 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:06.857091 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" path="/var/lib/kubelet/pods/8bfe3cdf-db10-474c-ab73-40bba8ab6b3a/volumes" Nov 03 22:34:13.141914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.141869 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:34:13.142463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.141955 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:34:13.185132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.185097 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x"] Nov 03 22:34:13.186247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.186220 4625 cpu_manager.go:414] "containerMap: removing container" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" Nov 03 22:34:13.186247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.186246 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" Nov 03 22:34:13.186474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.186314 4625 cpu_manager.go:414] "containerMap: removing container" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" Nov 03 22:34:13.186474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.186322 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" Nov 03 22:34:13.186584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.186501 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfe3cdf-db10-474c-ab73-40bba8ab6b3a" containerName="istio-proxy" Nov 03 22:34:13.186584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.186523 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b397c3-cce1-40d0-bd67-46c2b709bb99" containerName="istio-proxy" Nov 03 22:34:13.191652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.191618 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.214996 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.213730 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d8a3d1-serving-cert\"" Nov 03 22:34:13.214996 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.214125 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x"] Nov 03 22:34:13.353603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.353558 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949b744-8cff-4426-9f0d-0d3b36027091-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.353831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.353784 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.353894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.353835 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9qnl\" (UniqueName: \"kubernetes.io/projected/9949b744-8cff-4426-9f0d-0d3b36027091-kube-api-access-m9qnl\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.354064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.354040 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.470796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.458944 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.470796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.459293 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949b744-8cff-4426-9f0d-0d3b36027091-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.470796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.459393 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.470796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.459424 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9qnl\" (UniqueName: \"kubernetes.io/projected/9949b744-8cff-4426-9f0d-0d3b36027091-kube-api-access-m9qnl\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.470796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.460628 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949b744-8cff-4426-9f0d-0d3b36027091-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.488739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.488578 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.507738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.498182 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.507738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.505333 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9qnl\" (UniqueName: \"kubernetes.io/projected/9949b744-8cff-4426-9f0d-0d3b36027091-kube-api-access-m9qnl\") pod \"maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:13.526741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:13.525046 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:14.076822 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:34:14.076787 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9949b744_8cff_4426_9f0d_0d3b36027091.slice/crio-5b4678d5d89dfe3f7b9b91d7e9ae7b4cf06b0016c89ed33897e7ef4dc667881d WatchSource:0}: Error finding container 5b4678d5d89dfe3f7b9b91d7e9ae7b4cf06b0016c89ed33897e7ef4dc667881d: Status 404 returned error can't find the container with id 5b4678d5d89dfe3f7b9b91d7e9ae7b4cf06b0016c89ed33897e7ef4dc667881d Nov 03 22:34:14.078164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.078135 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x"] Nov 03 22:34:14.493122 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.492263 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" event={"ID":"9949b744-8cff-4426-9f0d-0d3b36027091","Type":"ContainerStarted","Data":"5b4678d5d89dfe3f7b9b91d7e9ae7b4cf06b0016c89ed33897e7ef4dc667881d"} Nov 03 22:34:14.507912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.498932 4625 generic.go:356] "Generic (PLEG): container finished" podID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerID="f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094" exitCode=0 Nov 03 22:34:14.507912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.499109 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" event={"ID":"7ceb83ca-1c71-40d1-a181-88d931f3538d","Type":"ContainerDied","Data":"f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094"} Nov 03 22:34:14.508207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.508117 4625 scope.go:122] "RemoveContainer" containerID="f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094" Nov 03 22:34:14.544742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.536507 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:14.544742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.536823 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:14.544742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:14.536900 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:15.712740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:15.709008 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" event={"ID":"9949b744-8cff-4426-9f0d-0d3b36027091","Type":"ContainerStarted","Data":"d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37"} Nov 03 22:34:15.712740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:15.710331 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:15.746014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:15.745134 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" event={"ID":"7ceb83ca-1c71-40d1-a181-88d931f3538d","Type":"ContainerStarted","Data":"63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8"} Nov 03 22:34:15.757898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:15.757756 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" podStartSLOduration=2.200898677 podStartE2EDuration="2.757734481s" totalImagesPullingTime="556.835804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:15.753179534 +0000 UTC m=+1277.182515529" watchObservedRunningTime="2026-11-03 22:34:15.757734481 +0000 UTC m=+1277.187070475" Nov 03 22:34:15.858803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:15.844361 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:34:15.858803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:15.856854 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:34:15.858803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:15.856912 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 22:34:16.838742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:16.837955 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:34:16.838742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:16.838046 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 22:34:17.828740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:17.828652 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:34:17.828953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:17.828754 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 22:34:18.829098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:18.829062 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:34:18.829568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:18.829122 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 22:34:19.829251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:19.829217 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:34:19.829797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:19.829280 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 22:34:20.829356 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:20.829323 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:34:20.829802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:20.829373 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 22:34:21.828972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:21.828936 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:34:21.829161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:21.829005 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 22:34:22.095106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.095074 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j"] Nov 03 22:34:22.099048 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.099027 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.117573 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.117539 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j"] Nov 03 22:34:22.122646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.122606 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4c5345-serving-cert\"" Nov 03 22:34:22.135108 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.135070 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-maas-api-tls\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.135449 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.135428 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-metrics-certs\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.135521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.135487 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv6k\" (UniqueName: \"kubernetes.io/projected/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-kube-api-access-wxv6k\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.135786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.135758 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.241330 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.241287 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv6k\" (UniqueName: \"kubernetes.io/projected/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-kube-api-access-wxv6k\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.241587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.241501 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.241587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.241568 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-maas-api-tls\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.241974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.241953 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-metrics-certs\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.242666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.242636 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.247523 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.247489 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-maas-api-tls\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.248586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.248555 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-metrics-certs\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.264068 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.264038 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv6k\" (UniqueName: \"kubernetes.io/projected/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-kube-api-access-wxv6k\") pod \"maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.414790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.414490 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:22.834208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.834128 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:34:22.834640 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.834610 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:34:22.835344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.835322 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 22:34:22.866425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.866385 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 22:34:22.909308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:22.909283 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j"] Nov 03 22:34:22.912131 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:34:22.912106 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f3d6e6_b86d_4dd0_bb87_ed6413ec9f12.slice/crio-9c4bb029cfc21c0380e2f1631edc81a35199e57f5cd2ec2a2c1c6ca43670efd5 WatchSource:0}: Error finding container 9c4bb029cfc21c0380e2f1631edc81a35199e57f5cd2ec2a2c1c6ca43670efd5: Status 404 returned error can't find the container with id 9c4bb029cfc21c0380e2f1631edc81a35199e57f5cd2ec2a2c1c6ca43670efd5 Nov 03 22:34:23.930797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:23.930767 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" event={"ID":"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12","Type":"ContainerStarted","Data":"53e45323d1c950deca985459e229ac904d8ceca2ef04f0a1cb1bb1534fde530a"} Nov 03 22:34:23.931240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:23.931225 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:23.931319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:23.931308 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" event={"ID":"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12","Type":"ContainerStarted","Data":"9c4bb029cfc21c0380e2f1631edc81a35199e57f5cd2ec2a2c1c6ca43670efd5"} Nov 03 22:34:23.975692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:23.975589 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" podStartSLOduration=1.496057231 podStartE2EDuration="1.975575075s" totalImagesPullingTime="479.517844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:23.969932492 +0000 UTC m=+1285.399268485" watchObservedRunningTime="2026-11-03 22:34:23.975575075 +0000 UTC m=+1285.404911107" Nov 03 22:34:28.647120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.647082 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl"] Nov 03 22:34:28.650984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.650962 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.677557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.675453 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-26f8ba-serving-cert\"" Nov 03 22:34:28.677557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.676908 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl"] Nov 03 22:34:28.742699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.742663 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.743034 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.743010 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.743334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.743315 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.743549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.743533 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwg9\" (UniqueName: \"kubernetes.io/projected/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-kube-api-access-jlwg9\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.867737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.862510 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.867737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.862635 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.867737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.862831 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.867737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.862921 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwg9\" (UniqueName: \"kubernetes.io/projected/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-kube-api-access-jlwg9\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.867737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.866548 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.890748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.884053 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.890748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.887965 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.933794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.922599 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwg9\" (UniqueName: \"kubernetes.io/projected/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-kube-api-access-jlwg9\") pod \"maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:28.981743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:28.978219 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:29.508052 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:34:29.507701 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b91c160_5f2d_43b6_bf6f_b0c593b1a3c9.slice/crio-38f89019a5aa9dd53f7b952c6c4e2cd13f673e5b6d29543a34dc49a562274b5f WatchSource:0}: Error finding container 38f89019a5aa9dd53f7b952c6c4e2cd13f673e5b6d29543a34dc49a562274b5f: Status 404 returned error can't find the container with id 38f89019a5aa9dd53f7b952c6c4e2cd13f673e5b6d29543a34dc49a562274b5f Nov 03 22:34:29.510554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:29.509891 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl"] Nov 03 22:34:30.104739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.102860 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:34:30.383815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.383783 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75"] Nov 03 22:34:30.384603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.384483 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" containerID="cri-o://b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a" gracePeriod=30 Nov 03 22:34:30.392870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.392836 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" event={"ID":"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9","Type":"ContainerStarted","Data":"38f89019a5aa9dd53f7b952c6c4e2cd13f673e5b6d29543a34dc49a562274b5f"} Nov 03 22:34:30.400671 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.400645 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn"] Nov 03 22:34:30.400998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.400947 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" containerID="cri-o://085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247" gracePeriod=30 Nov 03 22:34:30.409976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.408463 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd"] Nov 03 22:34:30.409976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:30.408659 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" containerID="cri-o://79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24" gracePeriod=30 Nov 03 22:34:31.461738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:31.458854 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" event={"ID":"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9","Type":"ContainerStarted","Data":"0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0"} Nov 03 22:34:31.461738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:31.460172 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:31.508778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:31.508645 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" podStartSLOduration=2.797915502 podStartE2EDuration="3.508625097s" totalImagesPullingTime="710.709595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:31.507161164 +0000 UTC m=+1292.936497206" watchObservedRunningTime="2026-11-03 22:34:31.508625097 +0000 UTC m=+1292.937961092" Nov 03 22:34:32.274199 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:32.274163 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:34:35.852468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:35.852437 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:34:35.857059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:35.857034 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:34:35.905408 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:35.905382 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030085 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-credential-socket\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030166 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-data\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030218 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-credential-socket\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030249 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-envoy\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030284 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-certs\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030327 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-token\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-token\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030354 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-data\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030374 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-credential-socket" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.030414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030404 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-token\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-token\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030422 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-certs\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030461 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-data\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030493 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-kube-api-access-shnbz\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-kube-api-access-shnbz\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030529 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-podinfo\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030551 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istiod-ca-cert\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030573 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f03d18-4be1-471b-94ae-811f8bcec9e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f03d18-4be1-471b-94ae-811f8bcec9e2-istiod-ca-cert\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030797 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-certs" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.030954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.030951 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-certs" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.031380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.031176 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-data" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.031380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.031246 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-data" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.031488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.031371 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istiod-ca-cert" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:36.031584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.031560 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-data" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.031681 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.031658 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-credential-socket" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.032067 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032036 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f03d18-4be1-471b-94ae-811f8bcec9e2-istiod-ca-cert" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:36.032181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032065 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istiod-ca-cert\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.032181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032111 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-certs\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.032295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032256 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-socket\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.032295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032285 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-kube-api-access-t7sqq\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-kube-api-access-t7sqq\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.032421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032326 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-socket\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.032421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032365 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-envoy\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.032421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032411 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-podinfo\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.032580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032444 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-credential-socket\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.032580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032489 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-podinfo\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.032580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032487 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istiod-ca-cert" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:36.032580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032541 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-kube-api-access-nc89k\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-kube-api-access-nc89k\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.032800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032612 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-socket\") pod \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\" (UID: \"00f8e9cc-c495-4291-86c5-d64db5dd7af9\") " Nov 03 22:34:36.032800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032644 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-token\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-token\") pod \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\" (UID: \"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20\") " Nov 03 22:34:36.032800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.032681 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-envoy\") pod \"27f03d18-4be1-471b-94ae-811f8bcec9e2\" (UID: \"27f03d18-4be1-471b-94ae-811f8bcec9e2\") " Nov 03 22:34:36.033726 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.033665 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-socket" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.036924 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.036954 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.036968 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.036984 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.036998 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037010 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037023 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037025 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-podinfo" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037035 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037070 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f03d18-4be1-471b-94ae-811f8bcec9e2-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037084 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037097 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.037177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037166 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-kube-api-access-nc89k" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "kube-api-access-nc89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:36.037926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037394 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-socket" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.037926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.037510 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-certs" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.038448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.038156 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-token" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:36.039126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.038891 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-podinfo" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:34:36.039316 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.039289 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-credential-socket" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.041599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.041568 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-kube-api-access-t7sqq" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "kube-api-access-t7sqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:36.041845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.041751 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-token" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:36.041939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.041915 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-socket" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.043458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.043366 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-envoy" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.043458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.043372 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-envoy" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.043458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.043400 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-token" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:36.043458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.043422 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-envoy" pod "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" (UID: "2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:36.043458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.043434 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-podinfo" pod "00f8e9cc-c495-4291-86c5-d64db5dd7af9" (UID: "00f8e9cc-c495-4291-86c5-d64db5dd7af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:34:36.043813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.043588 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-kube-api-access-shnbz" pod "27f03d18-4be1-471b-94ae-811f8bcec9e2" (UID: "27f03d18-4be1-471b-94ae-811f8bcec9e2"). InnerVolumeSpecName "kube-api-access-shnbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:36.140478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.140434 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.140766 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.140748 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.140874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.140859 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7sqq\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-kube-api-access-t7sqq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.140993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.140977 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141072 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141170 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141160 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141240 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141400 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141501 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc89k\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-kube-api-access-nc89k\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141619 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141744 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.141905 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.141892 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.142047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.142035 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f8e9cc-c495-4291-86c5-d64db5dd7af9-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.142131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.142122 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.142210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.142200 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shnbz\" (UniqueName: \"kubernetes.io/projected/27f03d18-4be1-471b-94ae-811f8bcec9e2-kube-api-access-shnbz\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.142332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.142320 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:34:36.760565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.758392 4625 generic.go:356] "Generic (PLEG): container finished" podID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerID="085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247" exitCode=0 Nov 03 22:34:36.760565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.758688 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" Nov 03 22:34:36.819987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.790507 4625 generic.go:356] "Generic (PLEG): container finished" podID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerID="b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a" exitCode=0 Nov 03 22:34:36.819987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.794683 4625 generic.go:356] "Generic (PLEG): container finished" podID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerID="79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24" exitCode=0 Nov 03 22:34:36.819987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.800471 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" Nov 03 22:34:36.819987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.802041 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" Nov 03 22:34:36.828796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.827589 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" event={"ID":"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20","Type":"ContainerDied","Data":"085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247"} Nov 03 22:34:36.828796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.827630 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn" event={"ID":"2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20","Type":"ContainerDied","Data":"4389d642ac9f50d8838bac656d0e6a9e6b4e2942e1eb83bdbe7a72cb783df512"} Nov 03 22:34:36.828796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.827646 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" event={"ID":"00f8e9cc-c495-4291-86c5-d64db5dd7af9","Type":"ContainerDied","Data":"b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a"} Nov 03 22:34:36.828796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.827685 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75" event={"ID":"00f8e9cc-c495-4291-86c5-d64db5dd7af9","Type":"ContainerDied","Data":"ef5472693d219363e64221aa89e5cba67a10b1e12e6151cbbc197e987bb5ff4a"} Nov 03 22:34:36.828796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.827706 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" event={"ID":"27f03d18-4be1-471b-94ae-811f8bcec9e2","Type":"ContainerDied","Data":"79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24"} Nov 03 22:34:36.828796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.827747 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd" event={"ID":"27f03d18-4be1-471b-94ae-811f8bcec9e2","Type":"ContainerDied","Data":"f0eb8d6dd1d83e3a5a8f43df7fa0f15e1233bcf6f832f881deebed3e8dbee8c6"} Nov 03 22:34:36.828796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.827776 4625 scope.go:122] "RemoveContainer" containerID="085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247" Nov 03 22:34:36.934901 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.934877 4625 scope.go:122] "RemoveContainer" containerID="085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247" Nov 03 22:34:36.935260 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:36.935234 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247\": container with ID starting with 085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247 not found: ID does not exist" containerID="085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247" Nov 03 22:34:36.935310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.935266 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247"} err="failed to get container status \"085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247\": rpc error: code = NotFound desc = could not find container \"085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247\": container with ID starting with 085c4dbacff03e90d62e6894982963e93bef9ad92ffdaf0070c4af5b2072b247 not found: ID does not exist" Nov 03 22:34:36.935310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.935289 4625 scope.go:122] "RemoveContainer" containerID="b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a" Nov 03 22:34:36.950193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.950166 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn"] Nov 03 22:34:36.954281 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.954101 4625 scope.go:122] "RemoveContainer" containerID="b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a" Nov 03 22:34:36.954605 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:36.954551 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a\": container with ID starting with b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a not found: ID does not exist" containerID="b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a" Nov 03 22:34:36.954697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.954616 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a"} err="failed to get container status \"b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a\": rpc error: code = NotFound desc = could not find container \"b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a\": container with ID starting with b7f11cc2d727d0f2f38cf0a223579d25a5e87826b2ee02c7d20c7269f15c812a not found: ID does not exist" Nov 03 22:34:36.954697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.954645 4625 scope.go:122] "RemoveContainer" containerID="79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24" Nov 03 22:34:36.959476 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.959450 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-fsnjn"] Nov 03 22:34:36.980212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.980184 4625 scope.go:122] "RemoveContainer" containerID="79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24" Nov 03 22:34:36.983204 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:36.983176 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24\": container with ID starting with 79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24 not found: ID does not exist" containerID="79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24" Nov 03 22:34:36.983687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.983655 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24"} err="failed to get container status \"79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24\": rpc error: code = NotFound desc = could not find container \"79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24\": container with ID starting with 79ae0eafce0a7f0802f8e7f01429e5a806a5dccee916ce291f30747f39e40e24 not found: ID does not exist" Nov 03 22:34:36.992042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.992014 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75"] Nov 03 22:34:36.999341 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:36.999306 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6wd75"] Nov 03 22:34:37.004110 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:34:37.004065 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f03d18_4be1_471b_94ae_811f8bcec9e2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f8e9cc_c495_4291_86c5_d64db5dd7af9.slice/crio-ef5472693d219363e64221aa89e5cba67a10b1e12e6151cbbc197e987bb5ff4a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d011ffd_3ab5_4b4a_b749_96a6a9d1eb20.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f03d18_4be1_471b_94ae_811f8bcec9e2.slice/crio-f0eb8d6dd1d83e3a5a8f43df7fa0f15e1233bcf6f832f881deebed3e8dbee8c6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d011ffd_3ab5_4b4a_b749_96a6a9d1eb20.slice/crio-4389d642ac9f50d8838bac656d0e6a9e6b4e2942e1eb83bdbe7a72cb783df512\": RecentStats: unable to find data in memory cache]" Nov 03 22:34:37.026517 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:37.026367 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd"] Nov 03 22:34:37.031778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:37.031744 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nckfd"] Nov 03 22:34:38.485826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:38.485794 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 22:34:38.737789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:38.735419 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" path="/var/lib/kubelet/pods/00f8e9cc-c495-4291-86c5-d64db5dd7af9/volumes" Nov 03 22:34:38.769741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:38.769459 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" path="/var/lib/kubelet/pods/27f03d18-4be1-471b-94ae-811f8bcec9e2/volumes" Nov 03 22:34:38.771852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:38.770978 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" path="/var/lib/kubelet/pods/2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20/volumes" Nov 03 22:34:45.905896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:45.905846 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:34:45.906389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:45.905937 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:34:49.186107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.186067 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75"] Nov 03 22:34:49.187288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187262 4625 cpu_manager.go:414] "containerMap: removing container" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" Nov 03 22:34:49.187288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187287 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" Nov 03 22:34:49.187469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187313 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" Nov 03 22:34:49.187469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187320 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" Nov 03 22:34:49.187469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187368 4625 cpu_manager.go:414] "containerMap: removing container" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" Nov 03 22:34:49.187469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187377 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" Nov 03 22:34:49.187676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187577 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d011ffd-3ab5-4b4a-b749-96a6a9d1eb20" containerName="istio-proxy" Nov 03 22:34:49.187676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187604 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f8e9cc-c495-4291-86c5-d64db5dd7af9" containerName="istio-proxy" Nov 03 22:34:49.187676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.187625 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f03d18-4be1-471b-94ae-811f8bcec9e2" containerName="istio-proxy" Nov 03 22:34:49.190694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.190671 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.207687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.206685 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-32fdf2-openshift-default-dockercfg-4nf2c\"" Nov 03 22:34:49.224518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.224485 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75"] Nov 03 22:34:49.251986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.251942 4625 generic.go:356] "Generic (PLEG): container finished" podID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerID="e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e" exitCode=0 Nov 03 22:34:49.252183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.252074 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" event={"ID":"b3ab7891-a909-4113-8881-0dd30efd2d0c","Type":"ContainerDied","Data":"e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e"} Nov 03 22:34:49.252581 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.252558 4625 scope.go:122] "RemoveContainer" containerID="e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e" Nov 03 22:34:49.255029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.254993 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:49.255141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.255075 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:49.255141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.255118 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.355524 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-token\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.355635 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-envoy\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.355686 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzml\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-kube-api-access-qqzml\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.359989 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-socket\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.360072 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-credential-socket\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.360103 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-certs\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.360161 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istiod-ca-cert\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.360192 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-data\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.360811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.360275 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-podinfo\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485293 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-token\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485390 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-envoy\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485447 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzml\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-kube-api-access-qqzml\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485656 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-socket\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485734 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-credential-socket\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485770 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-certs\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485831 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istiod-ca-cert\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485864 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-data\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.488813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.485934 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-podinfo\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.546863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.545866 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-envoy\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.546863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.546585 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-socket\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.581350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.568114 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-credential-socket\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.581350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.568938 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-certs\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.581350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.569640 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istiod-ca-cert\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.581350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.569918 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-data\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.598649 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.598592 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-podinfo\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.643742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.643645 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzml\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-kube-api-access-qqzml\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.760272 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.760180 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-token\") pod \"e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:49.809331 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:49.809049 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:50.265934 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:50.265894 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" event={"ID":"b3ab7891-a909-4113-8881-0dd30efd2d0c","Type":"ContainerStarted","Data":"efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9"} Nov 03 22:34:50.460824 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:50.460783 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75"] Nov 03 22:34:50.471434 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:50.471376 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:50.471643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:50.471470 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:50.471643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:50.471521 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:51.133742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.133013 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:34:51.145198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.144843 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:34:51.145198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.144913 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 22:34:51.276577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.276341 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" event={"ID":"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58","Type":"ContainerStarted","Data":"a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af"} Nov 03 22:34:51.276577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.276404 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" event={"ID":"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58","Type":"ContainerStarted","Data":"757aa17819955f24d13ea818ab0b67cb16706380d24f04107a434512c09c31a3"} Nov 03 22:34:51.324219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.324154 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podStartSLOduration=2.3241349270000002 podStartE2EDuration="2.324134927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:51.319020883 +0000 UTC m=+1312.748356903" watchObservedRunningTime="2026-11-03 22:34:51.324134927 +0000 UTC m=+1312.753470922" Nov 03 22:34:51.816032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.815994 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:51.821645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:51.821619 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:52.137742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:52.137066 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:34:52.137742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:52.137131 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 22:34:52.317821 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:52.317052 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:52.399872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:52.397249 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:34:53.145740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:53.132742 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:34:53.146799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:53.146745 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 22:34:54.137740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:54.135386 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:34:54.137740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:54.135467 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 22:34:55.143737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:55.139845 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:34:55.143737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:55.139909 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 22:34:56.008284 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.007604 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj"] Nov 03 22:34:56.012209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.012186 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.016779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.015907 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj"] Nov 03 22:34:56.029709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.029685 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bbfc2e-openshift-default-dockercfg-7ff7b\"" Nov 03 22:34:56.132468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.132434 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:34:56.160117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.160071 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 22:34:56.205418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205342 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-credential-socket\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.205613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205459 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-envoy\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.205613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205504 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istiod-ca-cert\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.205735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205673 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-token\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.205791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205758 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-socket\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.205837 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205789 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8k8\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-kube-api-access-ld8k8\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.205889 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205869 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-data\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.205941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205930 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-certs\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.206024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.205997 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-podinfo\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.347741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.316362 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-token\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.347741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.346271 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-socket\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.347741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.346343 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8k8\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-kube-api-access-ld8k8\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.347741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.347363 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-socket\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.347741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.347529 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-data\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.347741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.347654 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-certs\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.363773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.349000 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-certs\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.363773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.349411 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-data\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.383737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.382517 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-podinfo\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.383737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.382686 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-credential-socket\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.409735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.408173 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-credential-socket\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.409735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.408848 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-envoy\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.409735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.408907 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istiod-ca-cert\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:56.410058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:56.410012 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istiod-ca-cert\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:57.001332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.001281 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-token\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:57.001530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.001418 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-envoy\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:57.017741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.001851 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-podinfo\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:57.017741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.002355 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8k8\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-kube-api-access-ld8k8\") pod \"e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:57.168015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.158950 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:34:57.168015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.159012 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 22:34:57.228170 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.228140 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:57.611205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.611169 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj"] Nov 03 22:34:57.612767 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:34:57.612734 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fccf19_eb27_458f_add7_b7ea20d0bc2c.slice/crio-31db21ee29dd61ab096d461a38a253298404659a852df68656b03435aa456014 WatchSource:0}: Error finding container 31db21ee29dd61ab096d461a38a253298404659a852df68656b03435aa456014: Status 404 returned error can't find the container with id 31db21ee29dd61ab096d461a38a253298404659a852df68656b03435aa456014 Nov 03 22:34:57.616086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.616049 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:57.616189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.616141 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:57.616256 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:57.616191 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:34:58.084738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.069514 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" event={"ID":"32fccf19-eb27-458f-add7-b7ea20d0bc2c","Type":"ContainerStarted","Data":"dd8789936a2ef062e2c8fc7e7c582335999dcfc25716a8df70dfd976b53c9318"} Nov 03 22:34:58.084738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.069593 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" event={"ID":"32fccf19-eb27-458f-add7-b7ea20d0bc2c","Type":"ContainerStarted","Data":"31db21ee29dd61ab096d461a38a253298404659a852df68656b03435aa456014"} Nov 03 22:34:58.171789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.165271 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:34:58.171789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.167537 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:34:58.174794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.174763 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 22:34:58.187024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.186646 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podStartSLOduration=3.186626645 podStartE2EDuration="3.186626645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:58.175310274 +0000 UTC m=+1319.604646268" watchObservedRunningTime="2026-11-03 22:34:58.186626645 +0000 UTC m=+1319.615962640" Nov 03 22:34:58.228307 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.228275 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:34:58.230010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.229979 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:34:58.230247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:58.230220 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 22:34:59.234110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:34:59.234080 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:35:00.090808 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:00.090776 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:35:00.091779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:00.091757 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:35:02.188390 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:35:02.188349 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:35:05.046216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:05.046178 4625 scope.go:122] "RemoveContainer" containerID="c9a20b7cf2216fe3ef0ede86299f16ee6f4ac0a503c02c416036d41af22730e7" Nov 03 22:35:07.488378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.488340 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n"] Nov 03 22:35:07.494415 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.494392 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.521704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.521675 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-65b0e2-openshift-default-dockercfg-jtdc2\"" Nov 03 22:35:07.524625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.524598 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n"] Nov 03 22:35:07.570884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.570849 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-certs\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.571081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571067 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-envoy\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.571329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571305 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-data\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.571431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571351 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-socket\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.572222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571610 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-podinfo\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.572222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571678 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istiod-ca-cert\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.572222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571741 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-credential-socket\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.572222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571775 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-token\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.572222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.571816 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5f5z\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-kube-api-access-d5f5z\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.678981 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istiod-ca-cert\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679063 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-credential-socket\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679101 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-token\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679144 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5f5z\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-kube-api-access-d5f5z\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679224 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-certs\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679279 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-envoy\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679503 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-data\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679551 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-socket\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.679701 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-podinfo\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.687604 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istiod-ca-cert\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.696742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.687935 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-credential-socket\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.729167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.707658 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-data\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.729167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.707973 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-certs\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.729167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.708211 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-socket\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.751787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.742817 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-podinfo\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.751787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.743036 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-envoy\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.813738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.783378 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-token\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.813738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.784049 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5f5z\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-kube-api-access-d5f5z\") pod \"e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:07.840921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:07.828998 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:09.370776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:09.370741 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n"] Nov 03 22:35:09.374856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:09.374820 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:35:09.374974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:09.374936 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:35:09.375047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:09.375010 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:35:10.026811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:10.024091 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" event={"ID":"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8","Type":"ContainerStarted","Data":"739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392"} Nov 03 22:35:10.026811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:10.024172 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" event={"ID":"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8","Type":"ContainerStarted","Data":"a79ab46746393ebf5b4ee82c901f83abaea7c5063742c0ad1251310d4e516c3a"} Nov 03 22:35:10.130756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:10.119583 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podStartSLOduration=3.119562355 podStartE2EDuration="3.119562355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:10.099348618 +0000 UTC m=+1331.528684613" watchObservedRunningTime="2026-11-03 22:35:10.119562355 +0000 UTC m=+1331.548898349" Nov 03 22:35:10.849745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:10.844543 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:10.880739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:10.869066 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:11.082746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:11.074997 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:11.106792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:11.089042 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:35:11.250745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:11.248832 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.431835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:11.410853 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:35:11.953743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:11.952865 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.953743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:11.953234 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:15.824924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:15.824889 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5"] Nov 03 22:35:15.825422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:15.825084 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" containerID="cri-o://475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc" gracePeriod=30 Nov 03 22:35:15.836509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:15.836463 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs"] Nov 03 22:35:15.836852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:15.836704 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" containerID="cri-o://08e68b193f8db9b58dc28725062d445fd4584f61a4d4725c8a966b1217a6280f" gracePeriod=30 Nov 03 22:35:15.845215 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:15.845140 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d"] Nov 03 22:35:15.845386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:15.845360 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" containerID="cri-o://2c0bd70fd887127f8d8cca306a436b1df10eab1819d6a01dde8a377b46c74e47" gracePeriod=30 Nov 03 22:35:21.268920 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.268483 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:35:21.271686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.271656 4625 generic.go:356] "Generic (PLEG): container finished" podID="6791a532-f4ab-40e4-9f0c-39631d879947" containerID="2c0bd70fd887127f8d8cca306a436b1df10eab1819d6a01dde8a377b46c74e47" exitCode=0 Nov 03 22:35:21.272031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.271859 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" event={"ID":"6791a532-f4ab-40e4-9f0c-39631d879947","Type":"ContainerDied","Data":"2c0bd70fd887127f8d8cca306a436b1df10eab1819d6a01dde8a377b46c74e47"} Nov 03 22:35:21.274704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.273869 4625 generic.go:356] "Generic (PLEG): container finished" podID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerID="475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc" exitCode=0 Nov 03 22:35:21.274704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.274088 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" Nov 03 22:35:21.274704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.274228 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" event={"ID":"3aeb8542-d030-42da-a1fe-878fbc13bb73","Type":"ContainerDied","Data":"475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc"} Nov 03 22:35:21.274704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.274268 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5" event={"ID":"3aeb8542-d030-42da-a1fe-878fbc13bb73","Type":"ContainerDied","Data":"057d4641c34228730f6fb4dfaead0844e176c9b5634ffd5a9dd3b577223b8182"} Nov 03 22:35:21.274704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.274300 4625 scope.go:122] "RemoveContainer" containerID="475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc" Nov 03 22:35:21.277406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.277385 4625 generic.go:356] "Generic (PLEG): container finished" podID="f2671461-d398-44a6-a9c9-71917ce1d589" containerID="08e68b193f8db9b58dc28725062d445fd4584f61a4d4725c8a966b1217a6280f" exitCode=0 Nov 03 22:35:21.277573 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.277546 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" event={"ID":"f2671461-d398-44a6-a9c9-71917ce1d589","Type":"ContainerDied","Data":"08e68b193f8db9b58dc28725062d445fd4584f61a4d4725c8a966b1217a6280f"} Nov 03 22:35:21.298011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.297989 4625 scope.go:122] "RemoveContainer" containerID="475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc" Nov 03 22:35:21.298307 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:35:21.298286 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc\": container with ID starting with 475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc not found: ID does not exist" containerID="475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc" Nov 03 22:35:21.298383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.298318 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc"} err="failed to get container status \"475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc\": rpc error: code = NotFound desc = could not find container \"475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc\": container with ID starting with 475c3373d071b37222bab91c96e14b53ff438b4d7c1fa66098b378b370342efc not found: ID does not exist" Nov 03 22:35:21.323607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.323581 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-credential-socket\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.323782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.323634 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-podinfo\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.323782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.323700 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-envoy\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.323782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.323761 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-data\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.323969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.323802 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aeb8542-d030-42da-a1fe-878fbc13bb73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeb8542-d030-42da-a1fe-878fbc13bb73-istiod-ca-cert\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.323969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.323869 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-certs\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.324186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.324121 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-socket\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.324373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.324206 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-token\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-token\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.324373 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.324327 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-kube-api-access-zvrzl\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-kube-api-access-zvrzl\") pod \"3aeb8542-d030-42da-a1fe-878fbc13bb73\" (UID: \"3aeb8542-d030-42da-a1fe-878fbc13bb73\") " Nov 03 22:35:21.324929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.324903 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-data" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.325675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.325147 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-socket" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.325675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.325064 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-certs" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.325675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.325630 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.325675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.325645 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.325675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.325655 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.326153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.325695 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aeb8542-d030-42da-a1fe-878fbc13bb73-istiod-ca-cert" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:21.326153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.325761 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-credential-socket" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.330453 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.330424 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-envoy" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.351369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.351316 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-kube-api-access-zvrzl" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "kube-api-access-zvrzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.352071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.352026 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-token" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.353923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.353883 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-podinfo" pod "3aeb8542-d030-42da-a1fe-878fbc13bb73" (UID: "3aeb8542-d030-42da-a1fe-878fbc13bb73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:21.356999 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.356970 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:35:21.363035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.362957 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:35:21.427288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427213 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-envoy\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.427288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427259 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2671461-d398-44a6-a9c9-71917ce1d589-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2671461-d398-44a6-a9c9-71917ce1d589-istiod-ca-cert\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.427288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427287 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-data\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.427495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427322 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-istio-token\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-istio-token\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.427495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427447 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-certs\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.427495 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427478 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-credential-socket\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.427592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427497 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-data\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.427592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427525 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6791a532-f4ab-40e4-9f0c-39631d879947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791a532-f4ab-40e4-9f0c-39631d879947-istiod-ca-cert\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.427592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427557 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-socket\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.427692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427604 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-credential-socket\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.427692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427632 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-socket\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.427819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427777 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-istio-token\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-istio-token\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.427897 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427877 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2671461-d398-44a6-a9c9-71917ce1d589-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2671461-d398-44a6-a9c9-71917ce1d589-istio-podinfo\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.427960 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427939 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-kube-api-access-rm7vv\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-kube-api-access-rm7vv\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.428015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.427979 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6791a532-f4ab-40e4-9f0c-39631d879947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791a532-f4ab-40e4-9f0c-39631d879947-istio-podinfo\") pod \"6791a532-f4ab-40e4-9f0c-39631d879947\" (UID: \"6791a532-f4ab-40e4-9f0c-39631d879947\") " Nov 03 22:35:21.428070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.428016 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-certs\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.428070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.428025 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-certs" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.428185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.428143 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-envoy\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.428242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.428204 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-kube-api-access-whfv9\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-kube-api-access-whfv9\") pod \"f2671461-d398-44a6-a9c9-71917ce1d589\" (UID: \"f2671461-d398-44a6-a9c9-71917ce1d589\") " Nov 03 22:35:21.428592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.428559 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-data" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.428741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.428589 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-socket" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.429018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.428981 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2671461-d398-44a6-a9c9-71917ce1d589-istiod-ca-cert" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:21.429210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429188 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-credential-socket" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.429287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429265 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2671461-d398-44a6-a9c9-71917ce1d589-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429291 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429307 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429320 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429334 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrzl\" (UniqueName: \"kubernetes.io/projected/3aeb8542-d030-42da-a1fe-878fbc13bb73-kube-api-access-zvrzl\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429350 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429364 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429378 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429385 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-certs" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.429561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429400 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-data" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.429561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429393 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeb8542-d030-42da-a1fe-878fbc13bb73-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429432 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeb8542-d030-42da-a1fe-878fbc13bb73-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.429943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429617 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-credential-socket" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.429943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429638 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-socket" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.429943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.429823 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6791a532-f4ab-40e4-9f0c-39631d879947-istiod-ca-cert" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:21.431667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.431639 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-kube-api-access-whfv9" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "kube-api-access-whfv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.432144 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.432110 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6791a532-f4ab-40e4-9f0c-39631d879947-istio-podinfo" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:21.433631 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.433607 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-istio-token" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.440887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.440856 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2671461-d398-44a6-a9c9-71917ce1d589-istio-podinfo" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:21.441005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.440899 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-kube-api-access-rm7vv" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "kube-api-access-rm7vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.441417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.441396 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-istio-token" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.441677 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.441649 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-envoy" pod "6791a532-f4ab-40e4-9f0c-39631d879947" (UID: "6791a532-f4ab-40e4-9f0c-39631d879947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.442767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.442742 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-envoy" pod "f2671461-d398-44a6-a9c9-71917ce1d589" (UID: "f2671461-d398-44a6-a9c9-71917ce1d589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.531157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531127 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531154 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531164 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791a532-f4ab-40e4-9f0c-39631d879947-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531172 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531180 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531188 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531196 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2671461-d398-44a6-a9c9-71917ce1d589-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531204 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm7vv\" (UniqueName: \"kubernetes.io/projected/6791a532-f4ab-40e4-9f0c-39631d879947-kube-api-access-rm7vv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531212 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791a532-f4ab-40e4-9f0c-39631d879947-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531220 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531229 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2671461-d398-44a6-a9c9-71917ce1d589-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531237 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whfv9\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-kube-api-access-whfv9\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531244 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791a532-f4ab-40e4-9f0c-39631d879947-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.531488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.531252 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2671461-d398-44a6-a9c9-71917ce1d589-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.651869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.651835 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5"] Nov 03 22:35:21.657848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:21.657818 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-f72v5"] Nov 03 22:35:22.289384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.289302 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" Nov 03 22:35:22.289384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.289343 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d" event={"ID":"6791a532-f4ab-40e4-9f0c-39631d879947","Type":"ContainerDied","Data":"33b4ccb969495018643b5d5f2624ecb7646d274a9950a96f2f6472db8cf546c8"} Nov 03 22:35:22.289928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.289415 4625 scope.go:122] "RemoveContainer" containerID="2c0bd70fd887127f8d8cca306a436b1df10eab1819d6a01dde8a377b46c74e47" Nov 03 22:35:22.292819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.292791 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" event={"ID":"f2671461-d398-44a6-a9c9-71917ce1d589","Type":"ContainerDied","Data":"7b6f34d4645f34abe5bbb35d3d42d2353c42808ed4092a82b2ed19cbeb764be5"} Nov 03 22:35:22.293069 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.292890 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs" Nov 03 22:35:22.305123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.305104 4625 scope.go:122] "RemoveContainer" containerID="08e68b193f8db9b58dc28725062d445fd4584f61a4d4725c8a966b1217a6280f" Nov 03 22:35:22.362456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.362425 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d"] Nov 03 22:35:22.370746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.370700 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-94n6d"] Nov 03 22:35:22.397549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.397519 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs"] Nov 03 22:35:22.402368 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.402345 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-vm8xs"] Nov 03 22:35:22.730023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.729993 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" path="/var/lib/kubelet/pods/3aeb8542-d030-42da-a1fe-878fbc13bb73/volumes" Nov 03 22:35:22.730569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.730556 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" path="/var/lib/kubelet/pods/6791a532-f4ab-40e4-9f0c-39631d879947/volumes" Nov 03 22:35:22.731034 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:22.731021 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" path="/var/lib/kubelet/pods/f2671461-d398-44a6-a9c9-71917ce1d589/volumes" Nov 03 22:35:30.199382 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.199349 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6"] Nov 03 22:35:30.199837 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.199552 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" containerID="cri-o://62654581063edafb736044850dab8447c1996bcb9e53832c56693a92b411762d" gracePeriod=30 Nov 03 22:35:30.210653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.210618 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk"] Nov 03 22:35:30.210927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.210896 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" containerID="cri-o://96740c400e3fb90614082db87821e7c770ef1efcb53c0196728840e50ebf0e6c" gracePeriod=30 Nov 03 22:35:30.217117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.217084 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6"] Nov 03 22:35:30.217380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.217353 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" containerID="cri-o://2532559ed6388a5d360fbbb8dd3723f6bfaead745ad01c5ced4f0b062e60c9b7" gracePeriod=30 Nov 03 22:35:30.226083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.225687 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2"] Nov 03 22:35:30.226083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.225920 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" containerID="cri-o://733e9ad661b343ba6de35c74c814380734c6c2d265674ab58e39aac85c1e007b" gracePeriod=30 Nov 03 22:35:30.696784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.696749 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98"] Nov 03 22:35:30.697030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:30.697004 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" containerID="cri-o://6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52" gracePeriod=30 Nov 03 22:35:32.188997 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:35:32.188958 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:35:32.548798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:32.548670 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:35:32.548975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:32.548782 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:35:34.175728 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.175602 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj"] Nov 03 22:35:34.177688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.177648 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" Nov 03 22:35:34.177945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.177925 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" Nov 03 22:35:34.178060 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.178049 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" Nov 03 22:35:34.178149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.178138 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" Nov 03 22:35:34.178238 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.178229 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" Nov 03 22:35:34.178314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.178305 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" Nov 03 22:35:34.178580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.178567 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6791a532-f4ab-40e4-9f0c-39631d879947" containerName="istio-proxy" Nov 03 22:35:34.178676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.178665 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2671461-d398-44a6-a9c9-71917ce1d589" containerName="istio-proxy" Nov 03 22:35:34.178783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.178773 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aeb8542-d030-42da-a1fe-878fbc13bb73" containerName="istio-proxy" Nov 03 22:35:34.185199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.185175 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj"] Nov 03 22:35:34.185444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.185426 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.202079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.202057 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bbfc2e-serving-cert\"" Nov 03 22:35:34.285626 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.285588 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1db5e47-fa56-456a-9de4-bde181128810-openshift-service-ca\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.285868 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.285822 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p545s\" (UniqueName: \"kubernetes.io/projected/b1db5e47-fa56-456a-9de4-bde181128810-kube-api-access-p545s\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.285976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.285880 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-metrics-certs\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.286222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.286192 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-maas-api-tls\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.388171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.388133 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1db5e47-fa56-456a-9de4-bde181128810-openshift-service-ca\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.388379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.388281 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p545s\" (UniqueName: \"kubernetes.io/projected/b1db5e47-fa56-456a-9de4-bde181128810-kube-api-access-p545s\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.388379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.388322 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-metrics-certs\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.388586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.388567 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-maas-api-tls\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.388906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.388875 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1db5e47-fa56-456a-9de4-bde181128810-openshift-service-ca\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.391009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.390979 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-maas-api-tls\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.391106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.391081 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-metrics-certs\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.410409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.410373 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p545s\" (UniqueName: \"kubernetes.io/projected/b1db5e47-fa56-456a-9de4-bde181128810-kube-api-access-p545s\") pod \"maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:34.507787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:34.507047 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:35.010916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.010872 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj"] Nov 03 22:35:35.028744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.013089 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:35:35.762463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.761396 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" event={"ID":"b1db5e47-fa56-456a-9de4-bde181128810","Type":"ContainerStarted","Data":"347e96c3d91fd8d6d07d44b0b2273444d51f114b0581dfaae501387467615934"} Nov 03 22:35:35.762463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.762407 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:35.765402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.765225 4625 generic.go:356] "Generic (PLEG): container finished" podID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerID="62654581063edafb736044850dab8447c1996bcb9e53832c56693a92b411762d" exitCode=0 Nov 03 22:35:35.765402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.765310 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" event={"ID":"fb52bd42-200f-45fb-a87e-fd9b94b173b4","Type":"ContainerDied","Data":"62654581063edafb736044850dab8447c1996bcb9e53832c56693a92b411762d"} Nov 03 22:35:35.767978 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.767833 4625 generic.go:356] "Generic (PLEG): container finished" podID="ce339ce7-c721-474f-9f19-fabec26bffef" containerID="96740c400e3fb90614082db87821e7c770ef1efcb53c0196728840e50ebf0e6c" exitCode=0 Nov 03 22:35:35.767978 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.767930 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" event={"ID":"ce339ce7-c721-474f-9f19-fabec26bffef","Type":"ContainerDied","Data":"96740c400e3fb90614082db87821e7c770ef1efcb53c0196728840e50ebf0e6c"} Nov 03 22:35:35.770505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.770349 4625 generic.go:356] "Generic (PLEG): container finished" podID="f0ada526-3272-44d6-a5ba-276199cb022b" containerID="2532559ed6388a5d360fbbb8dd3723f6bfaead745ad01c5ced4f0b062e60c9b7" exitCode=0 Nov 03 22:35:35.770919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.770677 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" event={"ID":"f0ada526-3272-44d6-a5ba-276199cb022b","Type":"ContainerDied","Data":"2532559ed6388a5d360fbbb8dd3723f6bfaead745ad01c5ced4f0b062e60c9b7"} Nov 03 22:35:35.773653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.773631 4625 generic.go:356] "Generic (PLEG): container finished" podID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerID="733e9ad661b343ba6de35c74c814380734c6c2d265674ab58e39aac85c1e007b" exitCode=0 Nov 03 22:35:35.774108 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.774086 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" event={"ID":"f225f6da-aad0-43cb-9fe8-6fab90bbf69b","Type":"ContainerDied","Data":"733e9ad661b343ba6de35c74c814380734c6c2d265674ab58e39aac85c1e007b"} Nov 03 22:35:35.774261 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.774246 4625 scope.go:122] "RemoveContainer" containerID="252d83aac6558a86207e4b9d8a14ba113749662c3957dd83dbcc913a59e02bcd" Nov 03 22:35:35.921356 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.921122 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.975882 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-token\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-token\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.975950 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istiod-ca-cert\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.976039 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-podinfo\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.976173 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-certs\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.976291 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-credential-socket\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.976372 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-kube-api-access-v8965\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-kube-api-access-v8965\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.976430 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-socket\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.976577 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-data\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:35.977767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.976615 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-envoy\") pod \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\" (UID: \"fb52bd42-200f-45fb-a87e-fd9b94b173b4\") " Nov 03 22:35:36.020791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:35.995518 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:35:36.032778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.030029 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-socket" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.032778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.030332 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istiod-ca-cert" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.032778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.031689 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-data" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.043843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.035397 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" podStartSLOduration=1.5909065070000001 podStartE2EDuration="2.035377197s" totalImagesPullingTime="444.47069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:35.811804998 +0000 UTC m=+1357.241140991" watchObservedRunningTime="2026-11-03 22:35:36.035377197 +0000 UTC m=+1357.464713192" Nov 03 22:35:36.044578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.035703 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-certs" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.044578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.035917 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-credential-socket" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.099162 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-podinfo" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.100688 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-envoy\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.100790 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-certs\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.100944 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce339ce7-c721-474f-9f19-fabec26bffef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce339ce7-c721-474f-9f19-fabec26bffef-istiod-ca-cert\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.101011 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-socket\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.101243 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-istio-token\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-istio-token\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.101362 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce339ce7-c721-474f-9f19-fabec26bffef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce339ce7-c721-474f-9f19-fabec26bffef-istio-podinfo\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.101413 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-credential-socket\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.101551 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-data\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.109742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.101682 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-kube-api-access-klg4w\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-kube-api-access-klg4w\") pod \"ce339ce7-c721-474f-9f19-fabec26bffef\" (UID: \"ce339ce7-c721-474f-9f19-fabec26bffef\") " Nov 03 22:35:36.113741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.111705 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-socket" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.113741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.112254 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-certs" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.113741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.112652 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce339ce7-c721-474f-9f19-fabec26bffef-istiod-ca-cert" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.116146 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-token" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.116684 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-data" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.116762 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-credential-socket" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.119538 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-istio-token" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.120030 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-envoy" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.120128 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce339ce7-c721-474f-9f19-fabec26bffef-istio-podinfo" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.129377 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-kube-api-access-klg4w" pod "ce339ce7-c721-474f-9f19-fabec26bffef" (UID: "ce339ce7-c721-474f-9f19-fabec26bffef"). InnerVolumeSpecName "kube-api-access-klg4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.143923 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.143954 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.143972 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.143988 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klg4w\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-kube-api-access-klg4w\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144004 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144019 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144031 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144044 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144058 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce339ce7-c721-474f-9f19-fabec26bffef-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144072 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144082 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144090 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce339ce7-c721-474f-9f19-fabec26bffef-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144098 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144106 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144113 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce339ce7-c721-474f-9f19-fabec26bffef-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.165568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.144121 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce339ce7-c721-474f-9f19-fabec26bffef-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.168985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.168941 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-kube-api-access-v8965" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "kube-api-access-v8965". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.169576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.169535 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-envoy" pod "fb52bd42-200f-45fb-a87e-fd9b94b173b4" (UID: "fb52bd42-200f-45fb-a87e-fd9b94b173b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.227279 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.226925 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:35:36.249964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.249915 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8965\" (UniqueName: \"kubernetes.io/projected/fb52bd42-200f-45fb-a87e-fd9b94b173b4-kube-api-access-v8965\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.249964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.249949 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb52bd42-200f-45fb-a87e-fd9b94b173b4-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.307638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.307610 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:35:36.350698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.350661 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-kube-api-access-wp45x\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-kube-api-access-wp45x\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.351886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.351858 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-credential-socket" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.352259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352231 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-credential-socket\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.352670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352392 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-certs\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.352670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352521 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-envoy\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.352670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352555 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-podinfo\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.352670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352621 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-socket\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.352670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352662 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-token\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-token\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.353182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352704 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istiod-ca-cert\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.353182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.352785 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-data\") pod \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\" (UID: \"f225f6da-aad0-43cb-9fe8-6fab90bbf69b\") " Nov 03 22:35:36.354235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.354210 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.354448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.354424 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-data" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.354644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.354619 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-certs" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.355296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.355037 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-socket" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.356364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.356331 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istiod-ca-cert" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.357328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.357041 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-kube-api-access-wp45x" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "kube-api-access-wp45x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.358052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.358016 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-podinfo" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.360469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.360442 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-envoy" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.360571 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.360486 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-token" pod "f225f6da-aad0-43cb-9fe8-6fab90bbf69b" (UID: "f225f6da-aad0-43cb-9fe8-6fab90bbf69b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.364893 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.364869 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:35:36.455869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.455836 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0ada526-3272-44d6-a5ba-276199cb022b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ada526-3272-44d6-a5ba-276199cb022b-istio-podinfo\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.455994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.455941 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-istio-token\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-istio-token\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.455994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.455971 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-token\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-token\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456026 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c36897-4a20-4a43-a9b3-6874fe5af56e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c36897-4a20-4a43-a9b3-6874fe5af56e-istiod-ca-cert\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456094 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-credential-socket\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.456253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456133 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-certs\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456193 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ada526-3272-44d6-a5ba-276199cb022b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ada526-3272-44d6-a5ba-276199cb022b-istiod-ca-cert\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.456384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456309 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-envoy\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.456384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456350 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-envoy\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456383 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-credential-socket\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456459 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-certs\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.456584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456518 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-socket\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456546 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-data\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456639 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-podinfo\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456770 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-kube-api-access-lrf4m\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-kube-api-access-lrf4m\") pod \"16c36897-4a20-4a43-a9b3-6874fe5af56e\" (UID: \"16c36897-4a20-4a43-a9b3-6874fe5af56e\") " Nov 03 22:35:36.456897 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456883 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-socket\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.456989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.456974 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-data\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.457040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.457016 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-kube-api-access-kqwsf\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-kube-api-access-kqwsf\") pod \"f0ada526-3272-44d6-a5ba-276199cb022b\" (UID: \"f0ada526-3272-44d6-a5ba-276199cb022b\") " Nov 03 22:35:36.457959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.457927 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-data" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.458155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458133 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-credential-socket" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.458324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458263 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c36897-4a20-4a43-a9b3-6874fe5af56e-istiod-ca-cert" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.458531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458495 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458522 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458536 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458550 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458564 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458579 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458591 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458605 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458619 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp45x\" (UniqueName: \"kubernetes.io/projected/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-kube-api-access-wp45x\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.458680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458633 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225f6da-aad0-43cb-9fe8-6fab90bbf69b-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.459265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.458903 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-certs" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.459265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.459083 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-socket" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.459265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.459260 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-socket" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.461010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.460688 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ada526-3272-44d6-a5ba-276199cb022b-istiod-ca-cert" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.462004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.461416 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-credential-socket" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.462004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.461802 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-certs" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.462004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.461978 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-data" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.466943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.466903 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-token" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.467780 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.467742 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-envoy" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.467865 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.467811 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-istio-token" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.469247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.469219 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-kube-api-access-kqwsf" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "kube-api-access-kqwsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.472231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.472143 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-kube-api-access-lrf4m" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "kube-api-access-lrf4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.473292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.473250 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-envoy" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.474040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.474014 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0ada526-3272-44d6-a5ba-276199cb022b-istio-podinfo" pod "f0ada526-3272-44d6-a5ba-276199cb022b" (UID: "f0ada526-3272-44d6-a5ba-276199cb022b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.475303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.475281 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-podinfo" pod "16c36897-4a20-4a43-a9b3-6874fe5af56e" (UID: "16c36897-4a20-4a43-a9b3-6874fe5af56e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.561153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561051 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561095 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561110 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c36897-4a20-4a43-a9b3-6874fe5af56e-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561123 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561139 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561152 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ada526-3272-44d6-a5ba-276199cb022b-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561166 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561178 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561190 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561204 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c36897-4a20-4a43-a9b3-6874fe5af56e-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561220 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c36897-4a20-4a43-a9b3-6874fe5af56e-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561233 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrf4m\" (UniqueName: \"kubernetes.io/projected/16c36897-4a20-4a43-a9b3-6874fe5af56e-kube-api-access-lrf4m\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561245 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561261 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ada526-3272-44d6-a5ba-276199cb022b-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561274 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwsf\" (UniqueName: \"kubernetes.io/projected/f0ada526-3272-44d6-a5ba-276199cb022b-kube-api-access-kqwsf\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.561587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.561286 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ada526-3272-44d6-a5ba-276199cb022b-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.974183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.958451 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" event={"ID":"b1db5e47-fa56-456a-9de4-bde181128810","Type":"ContainerStarted","Data":"e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f"} Nov 03 22:35:36.974183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.961515 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" event={"ID":"fb52bd42-200f-45fb-a87e-fd9b94b173b4","Type":"ContainerDied","Data":"ef7c44268e918e1efcf652b018feeb89e85fd0a536345cafdfb7f85bea77fa1e"} Nov 03 22:35:36.974183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.961571 4625 scope.go:122] "RemoveContainer" containerID="62654581063edafb736044850dab8447c1996bcb9e53832c56693a92b411762d" Nov 03 22:35:36.974183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.961673 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6" Nov 03 22:35:36.995967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.995928 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" event={"ID":"ce339ce7-c721-474f-9f19-fabec26bffef","Type":"ContainerDied","Data":"ffe4d62a753f990e062d009376ddc9a9b4f585691cbd42d199bd8adf25bd5e3e"} Nov 03 22:35:36.996276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:36.996254 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk" Nov 03 22:35:37.002144 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.002122 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" Nov 03 22:35:37.002245 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.002120 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" event={"ID":"f0ada526-3272-44d6-a5ba-276199cb022b","Type":"ContainerDied","Data":"eee3854548b60d487fe1a1a251c0b7dc3a28d5072bb3d0fa26c874459a771909"} Nov 03 22:35:37.003934 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.003906 4625 generic.go:356] "Generic (PLEG): container finished" podID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerID="6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52" exitCode=0 Nov 03 22:35:37.004103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.004080 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" event={"ID":"16c36897-4a20-4a43-a9b3-6874fe5af56e","Type":"ContainerDied","Data":"6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52"} Nov 03 22:35:37.004171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.004130 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" event={"ID":"16c36897-4a20-4a43-a9b3-6874fe5af56e","Type":"ContainerDied","Data":"0e6d4d66dfd39be1ce11e53f3a34c2427b50e64bc8ce2cc972a02805a41f44b7"} Nov 03 22:35:37.004232 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.004207 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98" Nov 03 22:35:37.006538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.006503 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" Nov 03 22:35:37.006538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.006522 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2" event={"ID":"f225f6da-aad0-43cb-9fe8-6fab90bbf69b","Type":"ContainerDied","Data":"6a7f7940d1c7985dcb215b32e72a2eeeda9c1b46da48edad4dee0d7e7ed038a6"} Nov 03 22:35:37.011058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.011036 4625 scope.go:122] "RemoveContainer" containerID="96740c400e3fb90614082db87821e7c770ef1efcb53c0196728840e50ebf0e6c" Nov 03 22:35:37.022054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.022036 4625 scope.go:122] "RemoveContainer" containerID="2532559ed6388a5d360fbbb8dd3723f6bfaead745ad01c5ced4f0b062e60c9b7" Nov 03 22:35:37.033174 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.033154 4625 scope.go:122] "RemoveContainer" containerID="6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52" Nov 03 22:35:37.035094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.035070 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6"] Nov 03 22:35:37.044498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.044478 4625 scope.go:122] "RemoveContainer" containerID="6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52" Nov 03 22:35:37.044839 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:35:37.044819 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52\": container with ID starting with 6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52 not found: ID does not exist" containerID="6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52" Nov 03 22:35:37.044936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.044852 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52"} err="failed to get container status \"6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52\": rpc error: code = NotFound desc = could not find container \"6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52\": container with ID starting with 6c237e3fa33e2653a9d515af3935e707e709fc58f678a9a3c0dc23f92431ff52 not found: ID does not exist" Nov 03 22:35:37.044936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.044871 4625 scope.go:122] "RemoveContainer" containerID="733e9ad661b343ba6de35c74c814380734c6c2d265674ab58e39aac85c1e007b" Nov 03 22:35:37.046654 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.046632 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-jzgb6"] Nov 03 22:35:37.098295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.098258 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6"] Nov 03 22:35:37.105963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.105936 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6"] Nov 03 22:35:37.133574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.133542 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2"] Nov 03 22:35:37.138250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.138226 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-j5dc2"] Nov 03 22:35:37.169856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.169772 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98"] Nov 03 22:35:37.176057 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.176029 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-wmp98"] Nov 03 22:35:37.209956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.209876 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk"] Nov 03 22:35:37.223827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.223799 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-mn9sk"] Nov 03 22:35:37.249976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.249943 4625 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:35:37.250149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:37.249996 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-8l4s6" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:38.732081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:38.732047 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" path="/var/lib/kubelet/pods/16c36897-4a20-4a43-a9b3-6874fe5af56e/volumes" Nov 03 22:35:38.733028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:38.733006 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" path="/var/lib/kubelet/pods/ce339ce7-c721-474f-9f19-fabec26bffef/volumes" Nov 03 22:35:38.733687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:38.733668 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" path="/var/lib/kubelet/pods/f0ada526-3272-44d6-a5ba-276199cb022b/volumes" Nov 03 22:35:38.734310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:38.734294 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" path="/var/lib/kubelet/pods/f225f6da-aad0-43cb-9fe8-6fab90bbf69b/volumes" Nov 03 22:35:38.735101 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:38.735085 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" path="/var/lib/kubelet/pods/fb52bd42-200f-45fb-a87e-fd9b94b173b4/volumes" Nov 03 22:35:40.548835 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.548791 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9"] Nov 03 22:35:40.549963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.549941 4625 cpu_manager.go:414] "containerMap: removing container" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" Nov 03 22:35:40.550198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550174 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" Nov 03 22:35:40.550320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550224 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" Nov 03 22:35:40.550320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550233 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" Nov 03 22:35:40.550320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550262 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" Nov 03 22:35:40.550320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550270 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" Nov 03 22:35:40.550320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550291 4625 cpu_manager.go:414] "containerMap: removing container" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" Nov 03 22:35:40.550320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550299 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" Nov 03 22:35:40.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550330 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" Nov 03 22:35:40.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550337 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" Nov 03 22:35:40.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550530 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb52bd42-200f-45fb-a87e-fd9b94b173b4" containerName="istio-proxy" Nov 03 22:35:40.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550547 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" Nov 03 22:35:40.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550559 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce339ce7-c721-474f-9f19-fabec26bffef" containerName="istio-proxy" Nov 03 22:35:40.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550569 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c36897-4a20-4a43-a9b3-6874fe5af56e" containerName="istio-proxy" Nov 03 22:35:40.550664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.550583 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ada526-3272-44d6-a5ba-276199cb022b" containerName="istio-proxy" Nov 03 22:35:40.557841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.557816 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.585694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.585663 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9"] Nov 03 22:35:40.588617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.588572 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c0223-openshift-default-dockercfg-69t6f\"" Nov 03 22:35:40.646994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.646952 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3436b774-a017-4f06-866c-a48091023a5b-istiod-ca-cert\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647030 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-certs\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647118 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-data\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647163 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-credential-socket\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647310 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-istio-token\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647403 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-envoy\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647669 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647652 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3436b774-a017-4f06-866c-a48091023a5b-istio-podinfo\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647732 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pz7\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-kube-api-access-72pz7\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.647807 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.647770 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-socket\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.750559 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.750504 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3436b774-a017-4f06-866c-a48091023a5b-istio-podinfo\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.750886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.750862 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pz7\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-kube-api-access-72pz7\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.750911 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-socket\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.750946 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3436b774-a017-4f06-866c-a48091023a5b-istiod-ca-cert\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.751002 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-certs\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.751032 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-data\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.751067 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-credential-socket\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.751158 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-istio-token\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.751247 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-envoy\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.751755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.751730 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3436b774-a017-4f06-866c-a48091023a5b-istiod-ca-cert\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.754308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.751951 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-data\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.754452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.752134 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-credential-socket\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.754556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.752636 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-socket\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.754640 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.754248 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-certs\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.755423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.755389 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-envoy\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.787963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.787934 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3436b774-a017-4f06-866c-a48091023a5b-istio-podinfo\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.797167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.797110 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-istio-token\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.797693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.797665 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pz7\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-kube-api-access-72pz7\") pod \"e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:40.902126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:40.901257 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:42.384935 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:42.384896 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9"] Nov 03 22:35:42.389300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:42.389252 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:35:42.389412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:42.389346 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:35:42.389412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:42.389389 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:35:43.039757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:43.039709 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:35:43.366944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:43.366905 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" event={"ID":"3436b774-a017-4f06-866c-a48091023a5b","Type":"ContainerStarted","Data":"7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511"} Nov 03 22:35:43.367162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:43.366972 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" event={"ID":"3436b774-a017-4f06-866c-a48091023a5b","Type":"ContainerStarted","Data":"ef94acf39200361c48b75aeff96dc35a0731d5a0facbc9d16c39eaac8622312a"} Nov 03 22:35:43.414883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:43.414833 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podStartSLOduration=3.414819766 podStartE2EDuration="3.414819766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:43.411111596 +0000 UTC m=+1364.840447588" watchObservedRunningTime="2026-11-03 22:35:43.414819766 +0000 UTC m=+1364.844155827" Nov 03 22:35:43.904733 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:43.904389 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:43.909306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:43.909279 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:44.376579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:44.376543 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:44.377565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:44.377545 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:35:47.606441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.606407 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj"] Nov 03 22:35:47.607567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.607540 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" Nov 03 22:35:47.607567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.607564 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" Nov 03 22:35:47.607767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.607740 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f225f6da-aad0-43cb-9fe8-6fab90bbf69b" containerName="istio-proxy" Nov 03 22:35:47.612888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.612864 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.646313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.646264 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj"] Nov 03 22:35:47.646633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.646615 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-65b0e2-serving-cert\"" Nov 03 22:35:47.786969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.786936 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c978ef-76ac-44a5-9220-c574276181ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.787167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.786989 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-metrics-certs\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.787167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.787152 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-maas-api-tls\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.787286 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.787186 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57q5\" (UniqueName: \"kubernetes.io/projected/79c978ef-76ac-44a5-9220-c574276181ae-kube-api-access-w57q5\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.889164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.889067 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-maas-api-tls\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.889164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.889112 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57q5\" (UniqueName: \"kubernetes.io/projected/79c978ef-76ac-44a5-9220-c574276181ae-kube-api-access-w57q5\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.889535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.889507 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c978ef-76ac-44a5-9220-c574276181ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.889701 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.889662 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-metrics-certs\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.890251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.890225 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c978ef-76ac-44a5-9220-c574276181ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.892422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.892392 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-metrics-certs\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.892552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.892516 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-maas-api-tls\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.912082 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.912050 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57q5\" (UniqueName: \"kubernetes.io/projected/79c978ef-76ac-44a5-9220-c574276181ae-kube-api-access-w57q5\") pod \"maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:47.966736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:47.966028 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:49.030740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:49.020233 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj"] Nov 03 22:35:49.508117 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:49.508083 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" event={"ID":"79c978ef-76ac-44a5-9220-c574276181ae","Type":"ContainerStarted","Data":"07ca9cf20e79d2fe203ce1123fcc9abd186a0f5a28baba63896d8d5c1a192ebb"} Nov 03 22:35:49.508276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:49.508175 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" event={"ID":"79c978ef-76ac-44a5-9220-c574276181ae","Type":"ContainerStarted","Data":"f9c3b98ac5768b0e5d1aa4ab142b85808ddabff5c1ff70f667f78ed5005f87a2"} Nov 03 22:35:49.508276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:49.508245 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:35:49.559590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:49.559531 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" podStartSLOduration=2.184328074 podStartE2EDuration="2.559516523s" totalImagesPullingTime="375.188449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:49.554151459 +0000 UTC m=+1370.983487452" watchObservedRunningTime="2026-11-03 22:35:49.559516523 +0000 UTC m=+1370.988852515" Nov 03 22:35:55.522129 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:35:55.522098 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:36:00.511260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:00.511224 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v"] Nov 03 22:36:00.511654 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:00.511452 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" containerID="cri-o://286ba0710df0fb2e36aba63b4df4b4634a8996144bf1d3912497a50bd4c6aa57" gracePeriod=30 Nov 03 22:36:00.522243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:00.522213 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9"] Nov 03 22:36:00.522478 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:00.522451 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" containerID="cri-o://916eaed96bb6e0b1924de82cfe05748c725615dabd868bbd36635da75cff65db" gracePeriod=30 Nov 03 22:36:00.527382 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:00.527352 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6"] Nov 03 22:36:00.527569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:00.527534 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" containerID="cri-o://2d44c4e822a0567271e169c9273b1c2e8712b9bd1348d2b63774564565473146" gracePeriod=30 Nov 03 22:36:02.188312 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:36:02.188278 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:36:05.889015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.888617 4625 generic.go:356] "Generic (PLEG): container finished" podID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerID="2d44c4e822a0567271e169c9273b1c2e8712b9bd1348d2b63774564565473146" exitCode=0 Nov 03 22:36:05.889015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.888829 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" event={"ID":"47eef8a9-d72d-470d-9b3a-622ae9bf55e9","Type":"ContainerDied","Data":"2d44c4e822a0567271e169c9273b1c2e8712b9bd1348d2b63774564565473146"} Nov 03 22:36:05.891546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.891507 4625 generic.go:356] "Generic (PLEG): container finished" podID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerID="916eaed96bb6e0b1924de82cfe05748c725615dabd868bbd36635da75cff65db" exitCode=0 Nov 03 22:36:05.891676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.891647 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" event={"ID":"6afc35dd-60f0-43d0-a3b4-055daea16d82","Type":"ContainerDied","Data":"916eaed96bb6e0b1924de82cfe05748c725615dabd868bbd36635da75cff65db"} Nov 03 22:36:05.894222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.894133 4625 generic.go:356] "Generic (PLEG): container finished" podID="e504378b-605b-45b9-8573-33b540fafc36" containerID="286ba0710df0fb2e36aba63b4df4b4634a8996144bf1d3912497a50bd4c6aa57" exitCode=0 Nov 03 22:36:05.894222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.894192 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" event={"ID":"e504378b-605b-45b9-8573-33b540fafc36","Type":"ContainerDied","Data":"286ba0710df0fb2e36aba63b4df4b4634a8996144bf1d3912497a50bd4c6aa57"} Nov 03 22:36:05.920019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.919991 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:36:06.000054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:05.999858 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-podinfo\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000027 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-kube-api-access-sfgv8\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-kube-api-access-sfgv8\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000095 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-token\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-token\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000128 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-credential-socket\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000163 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-envoy\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000395 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-socket\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000603 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-certs\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000679 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000648 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6afc35dd-60f0-43d0-a3b4-055daea16d82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afc35dd-60f0-43d0-a3b4-055daea16d82-istiod-ca-cert\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.000748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.000699 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-data\") pod \"6afc35dd-60f0-43d0-a3b4-055daea16d82\" (UID: \"6afc35dd-60f0-43d0-a3b4-055daea16d82\") " Nov 03 22:36:06.008892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.004899 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-data" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.008892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.006307 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afc35dd-60f0-43d0-a3b4-055daea16d82-istiod-ca-cert" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:06.008892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.006629 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-socket" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.008892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.006970 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-credential-socket" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.009980 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.009911 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-kube-api-access-sfgv8" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "kube-api-access-sfgv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:06.010849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.010817 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-certs" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.013622 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.013597 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-envoy" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.020785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.016436 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-podinfo" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:06.020785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.017844 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-token" pod "6afc35dd-60f0-43d0-a3b4-055daea16d82" (UID: "6afc35dd-60f0-43d0-a3b4-055daea16d82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:06.086650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.086619 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:36:06.093421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.093396 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:36:06.103579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103553 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103580 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afc35dd-60f0-43d0-a3b4-055daea16d82-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103590 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103598 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103607 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfgv8\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-kube-api-access-sfgv8\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103615 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103625 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103635 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.103706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.103643 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afc35dd-60f0-43d0-a3b4-055daea16d82-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206005 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-envoy\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206069 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istiod-ca-cert\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206167 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-envoy\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206673 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e504378b-605b-45b9-8573-33b540fafc36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e504378b-605b-45b9-8573-33b540fafc36-istio-podinfo\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206748 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-data\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206806 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-certs\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206858 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-credential-socket\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.206931 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-credential-socket\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207018 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e504378b-605b-45b9-8573-33b540fafc36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e504378b-605b-45b9-8573-33b540fafc36-istiod-ca-cert\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207071 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-certs\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207114 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-kube-api-access-ldndg\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-kube-api-access-ldndg\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207161 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-socket\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207234 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-istio-token\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-istio-token\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207323 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-podinfo\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207361 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-data\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207423 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-socket\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207486 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-kube-api-access-xmd5d\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-kube-api-access-xmd5d\") pod \"e504378b-605b-45b9-8573-33b540fafc36\" (UID: \"e504378b-605b-45b9-8573-33b540fafc36\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207580 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-token\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-token\") pod \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\" (UID: \"47eef8a9-d72d-470d-9b3a-622ae9bf55e9\") " Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.207905 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istiod-ca-cert" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.208188 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-certs" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.208969 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-socket" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.210610 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-certs" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.210753 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-data" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.211025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.210962 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-credential-socket" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.212469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.211280 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e504378b-605b-45b9-8573-33b540fafc36-istiod-ca-cert" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:06.213132 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.213104 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-credential-socket" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.214683 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.214625 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-socket" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.214864 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-data" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.215916 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.215937 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.215951 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.215960 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.215970 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e504378b-605b-45b9-8573-33b540fafc36-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.215979 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.215991 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.216089 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e504378b-605b-45b9-8573-33b540fafc36-istio-podinfo" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.217201 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-istio-token" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:06.217606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.217273 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-podinfo" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:06.221560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.221266 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-kube-api-access-ldndg" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "kube-api-access-ldndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:06.221560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.221340 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-token" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:06.221560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.221348 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-envoy" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.221560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.221356 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-kube-api-access-xmd5d" pod "e504378b-605b-45b9-8573-33b540fafc36" (UID: "e504378b-605b-45b9-8573-33b540fafc36"). InnerVolumeSpecName "kube-api-access-xmd5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:06.221560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.221498 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-envoy" pod "47eef8a9-d72d-470d-9b3a-622ae9bf55e9" (UID: "47eef8a9-d72d-470d-9b3a-622ae9bf55e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:06.323623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323551 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldndg\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-kube-api-access-ldndg\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.323623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323601 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.323623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323617 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.323623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323631 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.324145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323645 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmd5d\" (UniqueName: \"kubernetes.io/projected/e504378b-605b-45b9-8573-33b540fafc36-kube-api-access-xmd5d\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.324145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323659 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.324145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323674 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.324145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323689 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e504378b-605b-45b9-8573-33b540fafc36-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.324145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323702 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e504378b-605b-45b9-8573-33b540fafc36-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.324145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323729 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.324145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.323744 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47eef8a9-d72d-470d-9b3a-622ae9bf55e9-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:06.903753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.903712 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" Nov 03 22:36:06.903753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.903725 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6" event={"ID":"47eef8a9-d72d-470d-9b3a-622ae9bf55e9","Type":"ContainerDied","Data":"0e97cd0bd2416fb319f31326d14414ad0ad07ef17b2fd3a2a83b96efad194d33"} Nov 03 22:36:06.904317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.903808 4625 scope.go:122] "RemoveContainer" containerID="2d44c4e822a0567271e169c9273b1c2e8712b9bd1348d2b63774564565473146" Nov 03 22:36:06.905321 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.905305 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" Nov 03 22:36:06.905432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.905345 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9" event={"ID":"6afc35dd-60f0-43d0-a3b4-055daea16d82","Type":"ContainerDied","Data":"aa543dc3707ffe82fb35ac65813c2f4ea5fb915cbc0472f87037dda35fbb4afd"} Nov 03 22:36:06.907423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.907397 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" event={"ID":"e504378b-605b-45b9-8573-33b540fafc36","Type":"ContainerDied","Data":"06deb7756bc926a2d5b5fb80914971d873a933e98bb63c3492c18b0459c64209"} Nov 03 22:36:06.907423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.907410 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v" Nov 03 22:36:06.917005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.916987 4625 scope.go:122] "RemoveContainer" containerID="916eaed96bb6e0b1924de82cfe05748c725615dabd868bbd36635da75cff65db" Nov 03 22:36:06.928578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.928557 4625 scope.go:122] "RemoveContainer" containerID="286ba0710df0fb2e36aba63b4df4b4634a8996144bf1d3912497a50bd4c6aa57" Nov 03 22:36:06.983027 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.982901 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v"] Nov 03 22:36:06.988652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:06.988623 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-5hj9v"] Nov 03 22:36:07.008339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:07.008304 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6"] Nov 03 22:36:07.014247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:07.014222 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-zglz6"] Nov 03 22:36:07.074400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:07.074366 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9"] Nov 03 22:36:07.082400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:07.082368 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-wtzl9"] Nov 03 22:36:08.731059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:08.731032 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" path="/var/lib/kubelet/pods/47eef8a9-d72d-470d-9b3a-622ae9bf55e9/volumes" Nov 03 22:36:08.731673 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:08.731657 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" path="/var/lib/kubelet/pods/6afc35dd-60f0-43d0-a3b4-055daea16d82/volumes" Nov 03 22:36:08.732254 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:08.732238 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e504378b-605b-45b9-8573-33b540fafc36" path="/var/lib/kubelet/pods/e504378b-605b-45b9-8573-33b540fafc36/volumes" Nov 03 22:36:11.883444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.883404 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56"] Nov 03 22:36:11.884563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884536 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" Nov 03 22:36:11.884563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884562 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" Nov 03 22:36:11.884892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884622 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" Nov 03 22:36:11.884892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884630 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" Nov 03 22:36:11.884892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884643 4625 cpu_manager.go:414] "containerMap: removing container" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" Nov 03 22:36:11.884892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884649 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" Nov 03 22:36:11.884892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884836 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e504378b-605b-45b9-8573-33b540fafc36" containerName="istio-proxy" Nov 03 22:36:11.884892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884858 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afc35dd-60f0-43d0-a3b4-055daea16d82" containerName="istio-proxy" Nov 03 22:36:11.884892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.884867 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="47eef8a9-d72d-470d-9b3a-622ae9bf55e9" containerName="istio-proxy" Nov 03 22:36:11.892506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.892479 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.912567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.912530 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b826b6-openshift-default-dockercfg-bjplt\"" Nov 03 22:36:11.918326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.918302 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56"] Nov 03 22:36:11.997166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997138 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-data\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997336 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997182 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-certs\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997356 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-token\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997410 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istiod-ca-cert\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997502 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-podinfo\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997564 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997537 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-credential-socket\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997801 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-socket\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997835 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcfd\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-kube-api-access-wbcfd\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:11.997916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:11.997897 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-envoy\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.109993 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-socket\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110045 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcfd\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-kube-api-access-wbcfd\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110081 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-envoy\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110200 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-data\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110243 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-certs\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110322 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-token\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110356 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istiod-ca-cert\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110413 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-podinfo\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.110439 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-credential-socket\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.111168 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-credential-socket\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.111747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.111405 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-socket\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.112505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.112201 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-data\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.112505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.112458 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-certs\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.113173 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.113145 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istiod-ca-cert\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.167799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.153701 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-podinfo\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.203890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.202829 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcfd\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-kube-api-access-wbcfd\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.224963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.204387 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-envoy\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.224963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.204967 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-token\") pod \"e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.224963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.214624 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:12.650558 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.650525 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56"] Nov 03 22:36:12.651185 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:36:12.651158 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3c54da_b7e6_4fdd_8a23_c424d582d8cb.slice/crio-317d5ba26ea8ea2a39d916c1ec935387f36ff01d34509d5a99407a5dfe327176 WatchSource:0}: Error finding container 317d5ba26ea8ea2a39d916c1ec935387f36ff01d34509d5a99407a5dfe327176: Status 404 returned error can't find the container with id 317d5ba26ea8ea2a39d916c1ec935387f36ff01d34509d5a99407a5dfe327176 Nov 03 22:36:12.654619 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.654586 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:12.654702 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.654648 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:12.654702 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:12.654676 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:13.022504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.022418 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" event={"ID":"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb","Type":"ContainerStarted","Data":"98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d"} Nov 03 22:36:13.022504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.022482 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" event={"ID":"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb","Type":"ContainerStarted","Data":"317d5ba26ea8ea2a39d916c1ec935387f36ff01d34509d5a99407a5dfe327176"} Nov 03 22:36:13.215149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.215114 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:13.289639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.289543 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:13.310906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.298889 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podStartSLOduration=2.298867038 podStartE2EDuration="2.298867038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:13.111964755 +0000 UTC m=+1394.541300746" watchObservedRunningTime="2026-11-03 22:36:13.298867038 +0000 UTC m=+1394.728203030" Nov 03 22:36:13.316306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.315851 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk"] Nov 03 22:36:13.330792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.328994 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.360095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.360062 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4b5e2b-serving-cert\"" Nov 03 22:36:13.423999 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.423965 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tqdv\" (UniqueName: \"kubernetes.io/projected/613cb909-ee79-4d35-ab13-e290d8736651-kube-api-access-7tqdv\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.424340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.424316 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-metrics-certs\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.424560 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.424546 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613cb909-ee79-4d35-ab13-e290d8736651-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.424746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.424732 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.466285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.466252 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk"] Nov 03 22:36:13.627740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.619917 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tqdv\" (UniqueName: \"kubernetes.io/projected/613cb909-ee79-4d35-ab13-e290d8736651-kube-api-access-7tqdv\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.627740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.620002 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-metrics-certs\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.627740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.620082 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613cb909-ee79-4d35-ab13-e290d8736651-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.627740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.620158 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.627740 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:36:13.620587 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-serving-cert: secret "maas-api-e2e-inf-a-4b5e2b-serving-cert" not found Nov 03 22:36:13.627740 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:36:13.620690 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls podName:613cb909-ee79-4d35-ab13-e290d8736651 nodeName:}" failed. No retries permitted until 2026-11-03 22:36:14.120658238 +0000 UTC m=+1395.549994226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls") pod "maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" (UID: "613cb909-ee79-4d35-ab13-e290d8736651") : secret "maas-api-e2e-inf-a-4b5e2b-serving-cert" not found Nov 03 22:36:13.640912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.633831 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613cb909-ee79-4d35-ab13-e290d8736651-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.681156 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.678916 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tqdv\" (UniqueName: \"kubernetes.io/projected/613cb909-ee79-4d35-ab13-e290d8736651-kube-api-access-7tqdv\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:13.688816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:13.686810 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-metrics-certs\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:14.033295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:14.033214 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:14.034161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:14.034136 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:36:14.136080 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:14.129471 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:14.139305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:14.139245 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls\") pod \"maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:14.336517 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:14.336421 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:14.851881 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:36:14.851844 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613cb909_ee79_4d35_ab13_e290d8736651.slice/crio-0bde2b3e861e04fff5c691770223b0c3ec29b54406ba6773cc66c8e1d2312003 WatchSource:0}: Error finding container 0bde2b3e861e04fff5c691770223b0c3ec29b54406ba6773cc66c8e1d2312003: Status 404 returned error can't find the container with id 0bde2b3e861e04fff5c691770223b0c3ec29b54406ba6773cc66c8e1d2312003 Nov 03 22:36:14.868047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:14.865081 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk"] Nov 03 22:36:15.054559 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:15.054521 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" event={"ID":"613cb909-ee79-4d35-ab13-e290d8736651","Type":"ContainerStarted","Data":"0bde2b3e861e04fff5c691770223b0c3ec29b54406ba6773cc66c8e1d2312003"} Nov 03 22:36:16.069459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:16.069427 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" event={"ID":"613cb909-ee79-4d35-ab13-e290d8736651","Type":"ContainerStarted","Data":"f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047"} Nov 03 22:36:16.069900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:16.069523 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:16.115054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:16.114994 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" podStartSLOduration=2.740350083 podStartE2EDuration="3.114975805s" totalImagesPullingTime="374.625722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:16.114195664 +0000 UTC m=+1397.543531656" watchObservedRunningTime="2026-11-03 22:36:16.114975805 +0000 UTC m=+1397.544311800" Nov 03 22:36:17.079989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:17.079954 4625 generic.go:356] "Generic (PLEG): container finished" podID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerID="a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af" exitCode=0 Nov 03 22:36:17.080488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:17.080024 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" event={"ID":"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58","Type":"ContainerDied","Data":"a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af"} Nov 03 22:36:17.080797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:17.080778 4625 scope.go:122] "RemoveContainer" containerID="a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af" Nov 03 22:36:17.082419 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:17.082389 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:17.082521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:17.082473 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:17.082588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:17.082520 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:18.321086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:18.319393 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" event={"ID":"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58","Type":"ContainerStarted","Data":"32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47"} Nov 03 22:36:18.816086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:18.816050 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:36:18.817033 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:18.817006 4625 patch_prober.go:29] interesting pod/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:18.817186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:18.817062 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 22:36:19.852893 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:19.850836 4625 patch_prober.go:29] interesting pod/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:19.852893 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:19.850898 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 22:36:20.817005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:20.816964 4625 patch_prober.go:29] interesting pod/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:20.817180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:20.817034 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 22:36:21.816783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:21.816747 4625 patch_prober.go:29] interesting pod/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:21.817208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:21.816811 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 22:36:22.086352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:22.086271 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:36:22.317023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:22.316986 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:36:22.816695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:22.816661 4625 patch_prober.go:29] interesting pod/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:22.816894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:22.816752 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 22:36:22.925598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:22.925550 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx"] Nov 03 22:36:22.947085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:22.947060 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:22.979767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:22.979324 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2c0223-serving-cert\"" Nov 03 22:36:23.004514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.004477 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx"] Nov 03 22:36:23.020171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.020118 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-metrics-certs\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.020328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.020269 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.020426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.020404 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkz5g\" (UniqueName: \"kubernetes.io/projected/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-kube-api-access-jkz5g\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.020631 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.020611 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-maas-api-tls\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.137737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.137133 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.137737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.137291 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkz5g\" (UniqueName: \"kubernetes.io/projected/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-kube-api-access-jkz5g\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.137737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.137423 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-maas-api-tls\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.137737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.137665 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-metrics-certs\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.162745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.157174 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.210391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.210355 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-metrics-certs\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.223821 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.223789 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-maas-api-tls\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.239242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.232272 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkz5g\" (UniqueName: \"kubernetes.io/projected/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-kube-api-access-jkz5g\") pod \"maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.313239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.313203 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:23.688289 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.688258 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx"] Nov 03 22:36:23.688832 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:36:23.688807 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0748a6_ff5f_4aec_bc2a_bd86149d1d9b.slice/crio-2fec86c1d44e29d4bbd0974ed756402ec4c60509e62f477aa010e22fae6b2d14 WatchSource:0}: Error finding container 2fec86c1d44e29d4bbd0974ed756402ec4c60509e62f477aa010e22fae6b2d14: Status 404 returned error can't find the container with id 2fec86c1d44e29d4bbd0974ed756402ec4c60509e62f477aa010e22fae6b2d14 Nov 03 22:36:23.822260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.822215 4625 patch_prober.go:29] interesting pod/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 22:36:23.823498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:23.822277 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 22:36:24.417407 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:24.417366 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" event={"ID":"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b","Type":"ContainerStarted","Data":"638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902"} Nov 03 22:36:24.417580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:24.417438 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" event={"ID":"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b","Type":"ContainerStarted","Data":"2fec86c1d44e29d4bbd0974ed756402ec4c60509e62f477aa010e22fae6b2d14"} Nov 03 22:36:24.418626 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:24.418602 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:24.464282 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:24.464215 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" podStartSLOduration=2.149023218 podStartE2EDuration="2.464195869s" totalImagesPullingTime="315.172651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:24.45968545 +0000 UTC m=+1405.889021443" watchObservedRunningTime="2026-11-03 22:36:24.464195869 +0000 UTC m=+1405.893531863" Nov 03 22:36:24.822451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:24.822359 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:36:24.823574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:24.823546 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:36:31.467976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:31.467940 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:36:31.800546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:31.800467 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs"] Nov 03 22:36:31.800740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:31.800671 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" podUID="0471975b-e9e0-41bc-af7a-b85fa6a87da2" containerName="maas-api" containerID="cri-o://9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b" gracePeriod=30 Nov 03 22:36:32.303738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.303603 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:36:32.341311 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.341265 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-maas-api-tls\") pod \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " Nov 03 22:36:32.341520 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.341390 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-metrics-certs\") pod \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " Nov 03 22:36:32.341589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.341519 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0471975b-e9e0-41bc-af7a-b85fa6a87da2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0471975b-e9e0-41bc-af7a-b85fa6a87da2-openshift-service-ca\") pod \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " Nov 03 22:36:32.341589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.341567 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0471975b-e9e0-41bc-af7a-b85fa6a87da2-kube-api-access-dm2b9\" (UniqueName: \"kubernetes.io/projected/0471975b-e9e0-41bc-af7a-b85fa6a87da2-kube-api-access-dm2b9\") pod \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\" (UID: \"0471975b-e9e0-41bc-af7a-b85fa6a87da2\") " Nov 03 22:36:32.348868 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.348819 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0471975b-e9e0-41bc-af7a-b85fa6a87da2-openshift-service-ca" pod "0471975b-e9e0-41bc-af7a-b85fa6a87da2" (UID: "0471975b-e9e0-41bc-af7a-b85fa6a87da2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:32.456362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.455789 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0471975b-e9e0-41bc-af7a-b85fa6a87da2-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:32.540981 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.540921 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0471975b-e9e0-41bc-af7a-b85fa6a87da2-kube-api-access-dm2b9" pod "0471975b-e9e0-41bc-af7a-b85fa6a87da2" (UID: "0471975b-e9e0-41bc-af7a-b85fa6a87da2"). InnerVolumeSpecName "kube-api-access-dm2b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:32.543925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.543858 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-metrics-certs" pod "0471975b-e9e0-41bc-af7a-b85fa6a87da2" (UID: "0471975b-e9e0-41bc-af7a-b85fa6a87da2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:32.548704 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:36:32.548553 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:36:32.552877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.552825 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-maas-api-tls" pod "0471975b-e9e0-41bc-af7a-b85fa6a87da2" (UID: "0471975b-e9e0-41bc-af7a-b85fa6a87da2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:32.568793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.564927 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:32.568793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.564969 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm2b9\" (UniqueName: \"kubernetes.io/projected/0471975b-e9e0-41bc-af7a-b85fa6a87da2-kube-api-access-dm2b9\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:32.568793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.564987 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0471975b-e9e0-41bc-af7a-b85fa6a87da2-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:36:32.978405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.977474 4625 generic.go:356] "Generic (PLEG): container finished" podID="0471975b-e9e0-41bc-af7a-b85fa6a87da2" containerID="9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b" exitCode=0 Nov 03 22:36:32.978405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.977750 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" Nov 03 22:36:32.988054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.988021 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" event={"ID":"0471975b-e9e0-41bc-af7a-b85fa6a87da2","Type":"ContainerDied","Data":"9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b"} Nov 03 22:36:32.988229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.988083 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs" event={"ID":"0471975b-e9e0-41bc-af7a-b85fa6a87da2","Type":"ContainerDied","Data":"e2ab5f14010f08e9847cab81f259a30850b0b5ce050ba015bfecd2944713332e"} Nov 03 22:36:32.988229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:32.988112 4625 scope.go:122] "RemoveContainer" containerID="9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b" Nov 03 22:36:33.003546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:33.003519 4625 scope.go:122] "RemoveContainer" containerID="9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b" Nov 03 22:36:33.003930 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:36:33.003904 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b\": container with ID starting with 9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b not found: ID does not exist" containerID="9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b" Nov 03 22:36:33.004007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:33.003939 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b"} err="failed to get container status \"9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b\": rpc error: code = NotFound desc = could not find container \"9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b\": container with ID starting with 9376ebe5f36387c4e9e4cd781b14b736fedaddadb5ef8d459b3249362875896b not found: ID does not exist" Nov 03 22:36:33.032381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:33.032351 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs"] Nov 03 22:36:33.043566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:33.043535 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-9mnjs"] Nov 03 22:36:34.744420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:34.744379 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0471975b-e9e0-41bc-af7a-b85fa6a87da2" path="/var/lib/kubelet/pods/0471975b-e9e0-41bc-af7a-b85fa6a87da2/volumes" Nov 03 22:36:51.777195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:51.777103 4625 generic.go:356] "Generic (PLEG): container finished" podID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerID="739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392" exitCode=0 Nov 03 22:36:51.777195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:51.777176 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" event={"ID":"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8","Type":"ContainerDied","Data":"739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392"} Nov 03 22:36:51.777839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:51.777799 4625 scope.go:122] "RemoveContainer" containerID="739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392" Nov 03 22:36:51.780273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:51.780236 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:51.780418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:51.780329 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:51.780418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:51.780388 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:36:52.791976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:52.791943 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" event={"ID":"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8","Type":"ContainerStarted","Data":"f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab"} Nov 03 22:36:52.846002 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:52.845906 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:36:52.867480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:52.867392 4625 patch_prober.go:29] interesting pod/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:36:52.867480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:52.867452 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 22:36:53.906751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:53.899795 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t"] Nov 03 22:36:53.940014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:53.939973 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0471975b-e9e0-41bc-af7a-b85fa6a87da2" containerName="maas-api" Nov 03 22:36:53.942216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:53.942178 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0471975b-e9e0-41bc-af7a-b85fa6a87da2" containerName="maas-api" Nov 03 22:36:53.942867 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:53.942848 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0471975b-e9e0-41bc-af7a-b85fa6a87da2" containerName="maas-api" Nov 03 22:36:54.047031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.026698 4625 patch_prober.go:29] interesting pod/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:36:54.047031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.026784 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 22:36:54.203778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.201638 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.209287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.205831 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t"] Nov 03 22:36:54.216197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.216169 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b826b6-serving-cert\"" Nov 03 22:36:54.304539 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.304506 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-metrics-certs\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.304890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.304868 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-maas-api-tls\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.305250 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.305225 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-openshift-service-ca\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.305388 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.305361 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4h2\" (UniqueName: \"kubernetes.io/projected/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-kube-api-access-gz4h2\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.406473 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.406434 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4h2\" (UniqueName: \"kubernetes.io/projected/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-kube-api-access-gz4h2\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.406695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.406629 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-metrics-certs\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.406695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.406689 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-maas-api-tls\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.406917 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.406898 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-openshift-service-ca\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.407462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.407437 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-openshift-service-ca\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.409352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.409327 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-metrics-certs\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.409486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.409467 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-maas-api-tls\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.429631 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.429606 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4h2\" (UniqueName: \"kubernetes.io/projected/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-kube-api-access-gz4h2\") pod \"maas-api-e2e-api-b-b826b6-687d77975b-hdq5t\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.527264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.527177 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:54.892762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.875279 4625 patch_prober.go:29] interesting pod/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:36:54.892762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:54.875339 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 22:36:55.180946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:55.180778 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t"] Nov 03 22:36:55.847539 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:55.844365 4625 patch_prober.go:29] interesting pod/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:36:55.847539 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:55.844438 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 22:36:56.056438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:56.056395 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:36:56.532805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:56.526072 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" event={"ID":"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060","Type":"ContainerStarted","Data":"ccdb6c1c7e9084106bb094c10562ea6695f789017b74b438ce859ba7941b9884"} Nov 03 22:36:56.532805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:56.526288 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" event={"ID":"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060","Type":"ContainerStarted","Data":"6a6f3215d8009d8b928e837a13eb4a6399bded6bb72ced3682e246bfc810a816"} Nov 03 22:36:56.532805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:56.527953 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:36:56.602352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:56.602296 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" podStartSLOduration=3.302377661 podStartE2EDuration="3.602281794s" totalImagesPullingTime="299.904133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:56.573832053 +0000 UTC m=+1438.003168046" watchObservedRunningTime="2026-11-03 22:36:56.602281794 +0000 UTC m=+1438.031617819" Nov 03 22:36:56.842795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:56.839067 4625 patch_prober.go:29] interesting pod/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:36:56.842795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:56.839124 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 22:36:57.856741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:57.856569 4625 patch_prober.go:29] interesting pod/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:36:57.856741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:57.856631 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 22:36:58.843742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:58.841612 4625 patch_prober.go:29] interesting pod/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:36:58.843742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:58.841669 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 22:36:59.815095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.815058 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn"] Nov 03 22:36:59.840738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.827088 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.850883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.850664 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn"] Nov 03 22:36:59.872313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.872271 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:36:59.873616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.873590 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:36:59.899165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.899135 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2d6076-openshift-default-dockercfg-wltsg\"" Nov 03 22:36:59.915406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.915373 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istiod-ca-cert\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.915582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.915442 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-data\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.915582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.915557 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-socket\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.915699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.915621 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfdj\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-kube-api-access-ckfdj\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.915810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.915772 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-credential-socket\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.915941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.915923 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-certs\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.916010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.915980 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-envoy\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.916052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.916028 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-token\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:36:59.916099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:36:59.916051 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-podinfo\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096220 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096175 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-certs\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096257 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-envoy\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096295 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-token\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096325 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-podinfo\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096436 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istiod-ca-cert\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096518 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-data\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096595 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-socket\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.096797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.096662 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfdj\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-kube-api-access-ckfdj\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.358277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.358191 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-credential-socket\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.433801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.366461 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istiod-ca-cert\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.434082 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.367121 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-certs\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.434211 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.399978 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-socket\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.435090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.435061 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-credential-socket\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.443312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.438705 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-data\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.454387 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.454353 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-token\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.481764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.477046 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-envoy\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.482658 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.482627 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-podinfo\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.494747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.483487 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfdj\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-kube-api-access-ckfdj\") pod \"e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:00.774474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:00.773069 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:01.112169 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:37:01.112138 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ff44b9_2e63_4538_a16f_0bdd609e16d7.slice/crio-6968c10cba5ed42446778dcbc0c4511d435adc2b71a656f24a1c9cf4cb04758f WatchSource:0}: Error finding container 6968c10cba5ed42446778dcbc0c4511d435adc2b71a656f24a1c9cf4cb04758f: Status 404 returned error can't find the container with id 6968c10cba5ed42446778dcbc0c4511d435adc2b71a656f24a1c9cf4cb04758f Nov 03 22:37:01.112609 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.112361 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn"] Nov 03 22:37:01.115209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.115175 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:01.115323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.115238 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:01.115323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.115266 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:01.676481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.676444 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" event={"ID":"e3ff44b9-2e63-4538-a16f-0bdd609e16d7","Type":"ContainerStarted","Data":"637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9"} Nov 03 22:37:01.676695 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.676498 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" event={"ID":"e3ff44b9-2e63-4538-a16f-0bdd609e16d7","Type":"ContainerStarted","Data":"6968c10cba5ed42446778dcbc0c4511d435adc2b71a656f24a1c9cf4cb04758f"} Nov 03 22:37:01.747966 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.747896 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podStartSLOduration=2.747877102 podStartE2EDuration="2.747877102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:01.736109326 +0000 UTC m=+1443.165445321" watchObservedRunningTime="2026-11-03 22:37:01.747877102 +0000 UTC m=+1443.177213096" Nov 03 22:37:01.775068 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:01.775030 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:02.188291 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:37:02.188255 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:37:02.542956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:02.542872 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:37:02.781331 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:02.781302 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:03.738748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:03.738380 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:03.758360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:03.758328 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:37:05.212452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:05.212417 4625 scope.go:122] "RemoveContainer" containerID="8621ca927abd21c9de46c72d69d7af394116de433eeec5b86642fc2cad036055" Nov 03 22:37:08.632319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.632286 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr"] Nov 03 22:37:08.640413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.640386 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.666292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.666262 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-22edc1-openshift-default-dockercfg-7cmfr\"" Nov 03 22:37:08.670052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.670014 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr"] Nov 03 22:37:08.714900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.714854 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-certs\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.714919 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istiod-ca-cert\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.715024 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-token\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.715236 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-podinfo\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.715337 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-credential-socket\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.715424 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-data\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715587 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.715571 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-envoy\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.715628 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-socket\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.715758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.715697 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrvq\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-kube-api-access-lwrvq\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.882487 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrvq\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-kube-api-access-lwrvq\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.882678 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-certs\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.882751 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istiod-ca-cert\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.882802 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-token\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.882960 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-podinfo\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.883037 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-credential-socket\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.883141 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-data\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.883231 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-envoy\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.883284 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-socket\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.883948 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-socket\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.884568 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-certs\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.885790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.885224 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istiod-ca-cert\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.887332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.887296 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-data\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.887794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.887774 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-credential-socket\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.927240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.927209 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-envoy\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:08.995063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:08.994991 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-podinfo\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:09.052736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:09.041644 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-token\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:09.052736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:09.041910 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrvq\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-kube-api-access-lwrvq\") pod \"e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:09.274126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:09.273980 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:09.629311 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:09.629280 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr"] Nov 03 22:37:09.629618 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:37:09.629595 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda812f72f_6465_4fcc_a0ed_e38e0925c91e.slice/crio-e60c3926488a4cab842c0fb4a90d10dcfc790682d0e23c9f69f4e5eab0c137d0 WatchSource:0}: Error finding container e60c3926488a4cab842c0fb4a90d10dcfc790682d0e23c9f69f4e5eab0c137d0: Status 404 returned error can't find the container with id e60c3926488a4cab842c0fb4a90d10dcfc790682d0e23c9f69f4e5eab0c137d0 Nov 03 22:37:09.632646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:09.632613 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:09.633002 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:09.632679 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:09.633002 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:09.632706 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:10.047020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:10.046922 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" event={"ID":"a812f72f-6465-4fcc-a0ed-e38e0925c91e","Type":"ContainerStarted","Data":"d875610069751dae6822eb42193b7d3ff5a295f65a73d1035887fbd3ce85d93a"} Nov 03 22:37:10.047020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:10.046982 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" event={"ID":"a812f72f-6465-4fcc-a0ed-e38e0925c91e","Type":"ContainerStarted","Data":"e60c3926488a4cab842c0fb4a90d10dcfc790682d0e23c9f69f4e5eab0c137d0"} Nov 03 22:37:10.293751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:10.293642 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:10.294061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:10.294039 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:37:10.308998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:10.308910 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:37:11.282792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:11.282757 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:11.343502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:11.343433 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podStartSLOduration=3.343411712 podStartE2EDuration="3.343411712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:10.120143552 +0000 UTC m=+1451.549479570" watchObservedRunningTime="2026-11-03 22:37:11.343411712 +0000 UTC m=+1452.772747707" Nov 03 22:37:12.067578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:12.067542 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:12.068620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:12.068598 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:37:13.375970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.375932 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss"] Nov 03 22:37:13.381234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.381209 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.398586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.396294 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5dd3b1-serving-cert\"" Nov 03 22:37:13.417502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.417464 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss"] Nov 03 22:37:13.521562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.521520 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989fm\" (UniqueName: \"kubernetes.io/projected/e3dfdae0-e574-4021-a94c-d2f62a81989a-kube-api-access-989fm\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.521785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.521763 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3dfdae0-e574-4021-a94c-d2f62a81989a-openshift-service-ca\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.521898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.521880 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-maas-api-tls\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.522076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.522058 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-metrics-certs\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.623666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.623622 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-metrics-certs\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.623918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.623833 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989fm\" (UniqueName: \"kubernetes.io/projected/e3dfdae0-e574-4021-a94c-d2f62a81989a-kube-api-access-989fm\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.623991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.623959 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3dfdae0-e574-4021-a94c-d2f62a81989a-openshift-service-ca\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.624072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.624057 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-maas-api-tls\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.624643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.624616 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3dfdae0-e574-4021-a94c-d2f62a81989a-openshift-service-ca\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.626314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.626243 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-metrics-certs\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.626438 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.626411 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-maas-api-tls\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.678044 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.676336 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-989fm\" (UniqueName: \"kubernetes.io/projected/e3dfdae0-e574-4021-a94c-d2f62a81989a-kube-api-access-989fm\") pod \"maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:13.739237 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:13.739190 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:14.799424 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:37:14.799282 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dfdae0_e574_4021_a94c_d2f62a81989a.slice/crio-149f8c538c58dbb11dcd8e4f48209f53697352afae366cd2be7dfb869ed83969 WatchSource:0}: Error finding container 149f8c538c58dbb11dcd8e4f48209f53697352afae366cd2be7dfb869ed83969: Status 404 returned error can't find the container with id 149f8c538c58dbb11dcd8e4f48209f53697352afae366cd2be7dfb869ed83969 Nov 03 22:37:14.805772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:14.805745 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss"] Nov 03 22:37:15.186534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:15.186499 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" event={"ID":"e3dfdae0-e574-4021-a94c-d2f62a81989a","Type":"ContainerStarted","Data":"149f8c538c58dbb11dcd8e4f48209f53697352afae366cd2be7dfb869ed83969"} Nov 03 22:37:16.256635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:16.256592 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" event={"ID":"e3dfdae0-e574-4021-a94c-d2f62a81989a","Type":"ContainerStarted","Data":"e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2"} Nov 03 22:37:16.263742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:16.258611 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:16.375742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:16.345030 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" podStartSLOduration=2.999806177 podStartE2EDuration="3.345007515s" totalImagesPullingTime="345.201338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:16.306667045 +0000 UTC m=+1457.736003039" watchObservedRunningTime="2026-11-03 22:37:16.345007515 +0000 UTC m=+1457.774343509" Nov 03 22:37:23.283744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:23.283681 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:37:28.014071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:28.014031 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j"] Nov 03 22:37:28.014927 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:28.014859 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" podUID="d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" containerName="maas-api" containerID="cri-o://53e45323d1c950deca985459e229ac904d8ceca2ef04f0a1cb1bb1534fde530a" gracePeriod=30 Nov 03 22:37:28.582749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:28.582081 4625 generic.go:356] "Generic (PLEG): container finished" podID="d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" containerID="53e45323d1c950deca985459e229ac904d8ceca2ef04f0a1cb1bb1534fde530a" exitCode=0 Nov 03 22:37:28.582749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:28.582265 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" event={"ID":"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12","Type":"ContainerDied","Data":"53e45323d1c950deca985459e229ac904d8ceca2ef04f0a1cb1bb1534fde530a"} Nov 03 22:37:29.441910 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.441880 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:37:29.593091 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.593059 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-kube-api-access-wxv6k\" (UniqueName: \"kubernetes.io/projected/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-kube-api-access-wxv6k\") pod \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " Nov 03 22:37:29.593292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.593164 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-openshift-service-ca\") pod \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " Nov 03 22:37:29.593292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.593231 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-maas-api-tls\") pod \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " Nov 03 22:37:29.593412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.593379 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-metrics-certs\") pod \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\" (UID: \"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12\") " Nov 03 22:37:29.597012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.596971 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-openshift-service-ca" pod "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" (UID: "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:29.630870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.625424 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-metrics-certs" pod "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" (UID: "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:29.630870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.625746 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-kube-api-access-wxv6k" pod "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" (UID: "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12"). InnerVolumeSpecName "kube-api-access-wxv6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:29.650949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.650904 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-maas-api-tls" pod "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" (UID: "d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:29.705744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.704912 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxv6k\" (UniqueName: \"kubernetes.io/projected/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-kube-api-access-wxv6k\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:37:29.705744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.704949 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:37:29.705744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.704963 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:37:29.705744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.704976 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:37:29.709049 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.709002 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" event={"ID":"d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12","Type":"ContainerDied","Data":"9c4bb029cfc21c0380e2f1631edc81a35199e57f5cd2ec2a2c1c6ca43670efd5"} Nov 03 22:37:29.709290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.709265 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j" Nov 03 22:37:29.709688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.709266 4625 scope.go:122] "RemoveContainer" containerID="53e45323d1c950deca985459e229ac904d8ceca2ef04f0a1cb1bb1534fde530a" Nov 03 22:37:29.800252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.799767 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j"] Nov 03 22:37:29.804110 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:29.804085 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c5345-cb76dd5f-dmw7j"] Nov 03 22:37:30.863736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:30.852731 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" path="/var/lib/kubelet/pods/d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12/volumes" Nov 03 22:37:32.326682 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:37:32.326640 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:37:38.042599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:38.042511 4625 generic.go:356] "Generic (PLEG): container finished" podID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerID="98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d" exitCode=0 Nov 03 22:37:38.042599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:38.042586 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" event={"ID":"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb","Type":"ContainerDied","Data":"98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d"} Nov 03 22:37:38.043153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:38.043136 4625 scope.go:122] "RemoveContainer" containerID="98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d" Nov 03 22:37:38.045199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:38.045166 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:38.045306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:38.045239 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:38.045306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:38.045274 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:39.056632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.056553 4625 generic.go:356] "Generic (PLEG): container finished" podID="3436b774-a017-4f06-866c-a48091023a5b" containerID="7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511" exitCode=0 Nov 03 22:37:39.056632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.056622 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" event={"ID":"3436b774-a017-4f06-866c-a48091023a5b","Type":"ContainerDied","Data":"7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511"} Nov 03 22:37:39.057552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.057221 4625 scope.go:122] "RemoveContainer" containerID="7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511" Nov 03 22:37:39.059447 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.059414 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:39.059581 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.059491 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:39.059581 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.059529 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:37:39.060031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.060005 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" event={"ID":"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb","Type":"ContainerStarted","Data":"3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440"} Nov 03 22:37:39.220066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.219977 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:37:39.236400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.236319 4625 patch_prober.go:29] interesting pod/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:37:39.236606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:39.236397 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 22:37:40.077811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:40.077769 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" event={"ID":"3436b774-a017-4f06-866c-a48091023a5b","Type":"ContainerStarted","Data":"573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7"} Nov 03 22:37:40.215243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:40.215204 4625 patch_prober.go:29] interesting pod/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:37:40.215422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:40.215254 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 22:37:40.902789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:40.902536 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:37:40.903880 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:40.903857 4625 patch_prober.go:29] interesting pod/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:37:40.903965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:40.903905 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 22:37:41.215381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:41.215298 4625 patch_prober.go:29] interesting pod/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:37:41.215381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:41.215364 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 22:37:41.903246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:41.903206 4625 patch_prober.go:29] interesting pod/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:37:41.903464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:41.903290 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 22:37:42.223802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:42.215779 4625 patch_prober.go:29] interesting pod/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:37:42.223802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:42.215835 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 22:37:42.903420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:42.903383 4625 patch_prober.go:29] interesting pod/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:37:42.903615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:42.903433 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 22:37:43.215831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:43.215412 4625 patch_prober.go:29] interesting pod/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:37:43.215831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:43.215470 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 22:37:43.910751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:43.908214 4625 patch_prober.go:29] interesting pod/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:37:43.910751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:43.908290 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 22:37:44.044742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:44.036803 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:37:44.215642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:44.215559 4625 patch_prober.go:29] interesting pod/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:37:44.215806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:44.215630 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 22:37:44.379741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:44.378835 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:37:44.903686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:44.903631 4625 patch_prober.go:29] interesting pod/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:37:44.903888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:44.903696 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 22:37:45.312681 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:45.312614 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:37:45.315249 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:45.315222 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:37:45.903557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:45.903526 4625 patch_prober.go:29] interesting pod/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 22:37:45.903769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:45.903578 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 22:37:46.908630 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:46.908599 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:37:46.909667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:37:46.909649 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:38:01.131875 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:01.131837 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:38:01.132390 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:01.131903 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:38:02.308102 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:38:02.307998 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:38:02.872400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:02.872351 4625 generic.go:356] "Generic (PLEG): container finished" podID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerID="dd8789936a2ef062e2c8fc7e7c582335999dcfc25716a8df70dfd976b53c9318" exitCode=0 Nov 03 22:38:02.872662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:02.872426 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" event={"ID":"32fccf19-eb27-458f-add7-b7ea20d0bc2c","Type":"ContainerDied","Data":"dd8789936a2ef062e2c8fc7e7c582335999dcfc25716a8df70dfd976b53c9318"} Nov 03 22:38:02.873038 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:02.873015 4625 scope.go:122] "RemoveContainer" containerID="dd8789936a2ef062e2c8fc7e7c582335999dcfc25716a8df70dfd976b53c9318" Nov 03 22:38:02.875673 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:02.875642 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:02.875820 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:02.875748 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:02.875820 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:02.875801 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:02.987408 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:02.987359 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:38:03.071672 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:03.071392 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:38:03.964760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:03.964412 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" event={"ID":"32fccf19-eb27-458f-add7-b7ea20d0bc2c","Type":"ContainerStarted","Data":"8c5e5bea7a6b493710a920e5349442d17fb3ff62b4af920ae54c82dd41c18fe2"} Nov 03 22:38:04.229212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:04.229132 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:38:04.229363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:04.229317 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:04.229458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:04.229354 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 22:38:05.228499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:05.228468 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:05.228915 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:05.228518 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 22:38:06.228686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:06.228649 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:06.229204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:06.228727 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 22:38:07.229347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:07.229310 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:07.229807 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:07.229362 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 22:38:08.228903 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:08.228870 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 03 22:38:08.229093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:08.228922 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 03 22:38:09.274852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:09.274820 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:38:09.276178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:09.276152 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:38:09.277351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:09.277329 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:38:18.611167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.611126 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn"] Nov 03 22:38:18.612355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.612329 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" containerName="maas-api" Nov 03 22:38:18.612355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.612352 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" containerName="maas-api" Nov 03 22:38:18.612574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.612559 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f3d6e6-b86d-4dd0-bb87-ed6413ec9f12" containerName="maas-api" Nov 03 22:38:18.616220 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.616199 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.637806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.637773 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8b1efeb6-openshift-default-dockercfg-c2bhg\"" Nov 03 22:38:18.708830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.708795 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn"] Nov 03 22:38:18.785939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.785286 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-data\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.785939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.785343 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.785939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.785378 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d50cd85-133d-447f-842c-b8debe5c5ac0-istiod-ca-cert\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.785939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.785431 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-podinfo\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.785939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.785542 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-credential-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.785939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.785588 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-certs\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.785939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.785660 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-token\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.787051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.786582 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-envoy\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.787051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.786639 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqf7f\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-kube-api-access-pqf7f\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.887830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.887743 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-credential-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.887830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.887800 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-certs\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.887849 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-token\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.887948 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-envoy\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.887989 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqf7f\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-kube-api-access-pqf7f\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888094 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-credential-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888186 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-data\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888203 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-certs\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888211 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888269 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d50cd85-133d-447f-842c-b8debe5c5ac0-istiod-ca-cert\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888316 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-podinfo\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888409 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.888554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.888516 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-data\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.889081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.889051 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d50cd85-133d-447f-842c-b8debe5c5ac0-istiod-ca-cert\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.890229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.890209 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-envoy\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.890884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.890864 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-podinfo\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.910949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.910925 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-token\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.911095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.911078 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqf7f\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-kube-api-access-pqf7f\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:18.930213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:18.930186 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:19.275810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:19.275772 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn"] Nov 03 22:38:19.276769 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:38:19.276739 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d50cd85_133d_447f_842c_b8debe5c5ac0.slice/crio-5ca5d3c345fc6aaac897aef9153738d2317898c3d62c27bc75c5bc3874e96036 WatchSource:0}: Error finding container 5ca5d3c345fc6aaac897aef9153738d2317898c3d62c27bc75c5bc3874e96036: Status 404 returned error can't find the container with id 5ca5d3c345fc6aaac897aef9153738d2317898c3d62c27bc75c5bc3874e96036 Nov 03 22:38:19.279886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:19.279851 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:19.280001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:19.279935 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:19.280001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:19.279994 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:20.292829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:20.292789 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" event={"ID":"2d50cd85-133d-447f-842c-b8debe5c5ac0","Type":"ContainerStarted","Data":"cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3"} Nov 03 22:38:20.293437 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:20.293412 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" event={"ID":"2d50cd85-133d-447f-842c-b8debe5c5ac0","Type":"ContainerStarted","Data":"5ca5d3c345fc6aaac897aef9153738d2317898c3d62c27bc75c5bc3874e96036"} Nov 03 22:38:20.367603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:20.367539 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podStartSLOduration=2.367518408 podStartE2EDuration="2.367518408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:20.350740072 +0000 UTC m=+1521.780076066" watchObservedRunningTime="2026-11-03 22:38:20.367518408 +0000 UTC m=+1521.796854401" Nov 03 22:38:20.932521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:20.930761 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:20.932833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:20.932468 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:38:20.933009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:20.932986 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 22:38:21.931380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:21.931345 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:38:21.931855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:21.931395 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 22:38:22.931403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:22.931375 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:38:22.931831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:22.931419 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 22:38:23.934741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:23.930848 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:38:23.934741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:23.930907 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 22:38:24.930822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:24.930793 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:38:24.931004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:24.930843 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 22:38:25.939544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:25.937181 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:38:25.939544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:25.937256 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 22:38:26.993742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:26.987273 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 22:38:26.993742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:26.987352 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 22:38:28.085429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:28.085395 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:28.087845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:28.087822 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:28.092517 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:28.092491 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:38:32.189401 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:38:32.189365 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:38:43.111913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:43.111864 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:38:43.112313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:43.111944 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:38:45.960645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:45.958896 4625 generic.go:356] "Generic (PLEG): container finished" podID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerID="637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9" exitCode=0 Nov 03 22:38:45.960645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:45.959133 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" event={"ID":"e3ff44b9-2e63-4538-a16f-0bdd609e16d7","Type":"ContainerDied","Data":"637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9"} Nov 03 22:38:45.960645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:45.959677 4625 scope.go:122] "RemoveContainer" containerID="637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9" Nov 03 22:38:45.975009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:45.973670 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:45.975009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:45.973813 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:45.975009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:45.973866 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:46.034039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:46.019298 4625 generic.go:356] "Generic (PLEG): container finished" podID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerID="d875610069751dae6822eb42193b7d3ff5a295f65a73d1035887fbd3ce85d93a" exitCode=0 Nov 03 22:38:46.034039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:46.019494 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" event={"ID":"a812f72f-6465-4fcc-a0ed-e38e0925c91e","Type":"ContainerDied","Data":"d875610069751dae6822eb42193b7d3ff5a295f65a73d1035887fbd3ce85d93a"} Nov 03 22:38:46.034039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:46.020117 4625 scope.go:122] "RemoveContainer" containerID="d875610069751dae6822eb42193b7d3ff5a295f65a73d1035887fbd3ce85d93a" Nov 03 22:38:46.282326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:46.282238 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:46.282623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:46.282597 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:46.282800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:46.282781 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:38:47.032227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.032140 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" event={"ID":"a812f72f-6465-4fcc-a0ed-e38e0925c91e","Type":"ContainerStarted","Data":"ba89f5a79c701766c9ac245ec06d016f9890c4578f6f9b5109835a1fd73eaba3"} Nov 03 22:38:47.034923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.034893 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" event={"ID":"e3ff44b9-2e63-4538-a16f-0bdd609e16d7","Type":"ContainerStarted","Data":"a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b"} Nov 03 22:38:47.275900 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.275856 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:38:47.277112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.277084 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:38:47.277277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.277139 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:38:47.775205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.775168 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:38:47.776099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.776062 4625 patch_prober.go:29] interesting pod/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 22:38:47.776242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:47.776131 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 22:38:48.280742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:48.276171 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:38:48.280742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:48.276234 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:38:48.730516 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:48.730490 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:38:48.783946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:48.783454 4625 patch_prober.go:29] interesting pod/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 22:38:48.783946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:48.783535 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 22:38:49.278746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:49.278119 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:38:49.278746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:49.278182 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:38:49.779741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:49.779091 4625 patch_prober.go:29] interesting pod/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 22:38:49.779741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:49.779159 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 22:38:50.285740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:50.281208 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:38:50.285740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:50.281276 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:38:50.788739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:50.777147 4625 patch_prober.go:29] interesting pod/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 22:38:50.788739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:50.777213 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 22:38:51.284511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:51.284472 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:38:51.284840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:51.284796 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:38:51.783829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:51.782842 4625 patch_prober.go:29] interesting pod/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 22:38:51.783829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:51.782907 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 22:38:52.276009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:52.275973 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:38:52.276644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:52.276597 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:38:52.482940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:52.482894 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:38:52.779739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:52.779148 4625 patch_prober.go:29] interesting pod/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 22:38:52.779739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:52.779221 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 22:38:53.280515 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:53.280473 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 22:38:53.281926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:53.281893 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 22:38:53.777285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:53.776601 4625 patch_prober.go:29] interesting pod/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 22:38:53.777285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:53.776668 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 22:38:54.183277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.183233 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75"] Nov 03 22:38:54.183525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.183500 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" containerID="cri-o://32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47" gracePeriod=30 Nov 03 22:38:54.292186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.292064 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:38:54.295791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.293950 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:38:54.333016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.332980 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:38:54.806147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.806111 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:38:54.808872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.808848 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:38:54.831568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.830325 4625 patch_prober.go:29] interesting pod/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:38:54.831568 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:54.830400 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:38:56.700769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:56.700698 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj"] Nov 03 22:38:56.701303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:56.701018 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" podUID="b1db5e47-fa56-456a-9de4-bde181128810" containerName="maas-api" containerID="cri-o://e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f" gracePeriod=30 Nov 03 22:38:57.237935 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.237903 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:38:57.367296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.367254 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-maas-api-tls\") pod \"b1db5e47-fa56-456a-9de4-bde181128810\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " Nov 03 22:38:57.370740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.367661 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1db5e47-fa56-456a-9de4-bde181128810-kube-api-access-p545s\" (UniqueName: \"kubernetes.io/projected/b1db5e47-fa56-456a-9de4-bde181128810-kube-api-access-p545s\") pod \"b1db5e47-fa56-456a-9de4-bde181128810\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " Nov 03 22:38:57.386010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.385971 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1db5e47-fa56-456a-9de4-bde181128810-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1db5e47-fa56-456a-9de4-bde181128810-openshift-service-ca\") pod \"b1db5e47-fa56-456a-9de4-bde181128810\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " Nov 03 22:38:57.397737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.386455 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-metrics-certs\") pod \"b1db5e47-fa56-456a-9de4-bde181128810\" (UID: \"b1db5e47-fa56-456a-9de4-bde181128810\") " Nov 03 22:38:57.457472 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.448939 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1db5e47-fa56-456a-9de4-bde181128810-openshift-service-ca" pod "b1db5e47-fa56-456a-9de4-bde181128810" (UID: "b1db5e47-fa56-456a-9de4-bde181128810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:57.457705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.455484 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-metrics-certs" pod "b1db5e47-fa56-456a-9de4-bde181128810" (UID: "b1db5e47-fa56-456a-9de4-bde181128810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:57.588766 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.564867 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:57.588766 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.564913 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1db5e47-fa56-456a-9de4-bde181128810-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:57.659897 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.639697 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-maas-api-tls" pod "b1db5e47-fa56-456a-9de4-bde181128810" (UID: "b1db5e47-fa56-456a-9de4-bde181128810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:57.698983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.698944 4625 generic.go:356] "Generic (PLEG): container finished" podID="b1db5e47-fa56-456a-9de4-bde181128810" containerID="e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f" exitCode=0 Nov 03 22:38:57.699434 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.699403 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" event={"ID":"b1db5e47-fa56-456a-9de4-bde181128810","Type":"ContainerDied","Data":"e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f"} Nov 03 22:38:57.704909 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.704870 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" event={"ID":"b1db5e47-fa56-456a-9de4-bde181128810","Type":"ContainerDied","Data":"347e96c3d91fd8d6d07d44b0b2273444d51f114b0581dfaae501387467615934"} Nov 03 22:38:57.708441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.705676 4625 scope.go:122] "RemoveContainer" containerID="e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f" Nov 03 22:38:57.708929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.708908 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj" Nov 03 22:38:57.711032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.701031 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1db5e47-fa56-456a-9de4-bde181128810-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:57.716656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.716608 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1db5e47-fa56-456a-9de4-bde181128810-kube-api-access-p545s" pod "b1db5e47-fa56-456a-9de4-bde181128810" (UID: "b1db5e47-fa56-456a-9de4-bde181128810"). InnerVolumeSpecName "kube-api-access-p545s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:57.813892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.813812 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p545s\" (UniqueName: \"kubernetes.io/projected/b1db5e47-fa56-456a-9de4-bde181128810-kube-api-access-p545s\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:57.831120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.831089 4625 scope.go:122] "RemoveContainer" containerID="e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f" Nov 03 22:38:57.841368 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:38:57.841324 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f\": container with ID starting with e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f not found: ID does not exist" containerID="e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f" Nov 03 22:38:57.841734 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:57.841689 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f"} err="failed to get container status \"e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f\": rpc error: code = NotFound desc = could not find container \"e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f\": container with ID starting with e9a6afc0f36e01383dd3de6f38adfbb98d1cbda860da6fc1a3c03898b472053f not found: ID does not exist" Nov 03 22:38:58.070275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:58.070241 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj"] Nov 03 22:38:58.078109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:58.078065 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbfc2e-778d6754c6-cs4pj"] Nov 03 22:38:58.733607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:58.733571 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1db5e47-fa56-456a-9de4-bde181128810" path="/var/lib/kubelet/pods/b1db5e47-fa56-456a-9de4-bde181128810/volumes" Nov 03 22:38:59.517147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.517118 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:38:59.557949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.557914 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-credential-socket\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.557982 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-data\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558049 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istiod-ca-cert\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558103 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-podinfo\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558232 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-token\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-token\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558229 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-data" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:59.558386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558240 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-credential-socket" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:59.558435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558384 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-kube-api-access-qqzml\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-kube-api-access-qqzml\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558494 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istiod-ca-cert" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:59.558579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558535 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-socket\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558611 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-certs\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558645 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-envoy\") pod \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\" (UID: \"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58\") " Nov 03 22:38:59.558984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558952 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-socket" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:59.559092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.558985 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-certs" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:59.559593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.559568 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.559704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.559595 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.559704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.559609 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.559704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.559624 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.559704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.559637 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.560559 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.560533 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-podinfo" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:59.560686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.560665 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-kube-api-access-qqzml" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "kube-api-access-qqzml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:59.560992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.560967 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-envoy" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:59.561218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.561192 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-token" pod "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" (UID: "1cabdb81-bea7-4c0d-8d1d-8f982ffffb58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:59.699089 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.699050 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.699345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.699308 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqzml\" (UniqueName: \"kubernetes.io/projected/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-kube-api-access-qqzml\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.699345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.699335 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.699345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.699352 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.787308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.787225 4625 generic.go:356] "Generic (PLEG): container finished" podID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerID="32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47" exitCode=0 Nov 03 22:38:59.787308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.787274 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" Nov 03 22:38:59.787711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.787327 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" event={"ID":"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58","Type":"ContainerDied","Data":"32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47"} Nov 03 22:38:59.787711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.787394 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75" event={"ID":"1cabdb81-bea7-4c0d-8d1d-8f982ffffb58","Type":"ContainerDied","Data":"757aa17819955f24d13ea818ab0b67cb16706380d24f04107a434512c09c31a3"} Nov 03 22:38:59.787711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.787426 4625 scope.go:122] "RemoveContainer" containerID="32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47" Nov 03 22:38:59.801845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.801824 4625 scope.go:122] "RemoveContainer" containerID="a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af" Nov 03 22:38:59.812963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.812939 4625 scope.go:122] "RemoveContainer" containerID="32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47" Nov 03 22:38:59.813257 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:38:59.813230 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47\": container with ID starting with 32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47 not found: ID does not exist" containerID="32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47" Nov 03 22:38:59.813333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.813259 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47"} err="failed to get container status \"32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47\": rpc error: code = NotFound desc = could not find container \"32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47\": container with ID starting with 32b8d0100947d070ad3dc74ac70b9a4f916124b0505752cf38f5a6dd15843d47 not found: ID does not exist" Nov 03 22:38:59.813333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.813277 4625 scope.go:122] "RemoveContainer" containerID="a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af" Nov 03 22:38:59.813551 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:38:59.813531 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af\": container with ID starting with a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af not found: ID does not exist" containerID="a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af" Nov 03 22:38:59.813607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.813557 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af"} err="failed to get container status \"a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af\": rpc error: code = NotFound desc = could not find container \"a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af\": container with ID starting with a7a0e95633435fb4467b95987aea13e59d61996fbb7e1c78b8cdcaa2cf13f2af not found: ID does not exist" Nov 03 22:38:59.844732 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.844683 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75"] Nov 03 22:38:59.850076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:38:59.850047 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-32fdf2-openshift-default-798db88bbc-52z75"] Nov 03 22:39:00.730166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:00.730133 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" path="/var/lib/kubelet/pods/1cabdb81-bea7-4c0d-8d1d-8f982ffffb58/volumes" Nov 03 22:39:02.187858 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:02.187817 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:39:05.079707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:05.079633 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj"] Nov 03 22:39:05.080233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:05.079914 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" containerID="cri-o://8c5e5bea7a6b493710a920e5349442d17fb3ff62b4af920ae54c82dd41c18fe2" gracePeriod=30 Nov 03 22:39:09.278301 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:09.278260 4625 patch_prober.go:29] interesting pod/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:39:09.278688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:09.278352 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:39:09.771855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:09.771820 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj"] Nov 03 22:39:09.772303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:09.772258 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" podUID="79c978ef-76ac-44a5-9220-c574276181ae" containerName="maas-api" containerID="cri-o://07ca9cf20e79d2fe203ce1123fcc9abd186a0f5a28baba63896d8d5c1a192ebb" gracePeriod=30 Nov 03 22:39:10.003677 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.003637 4625 generic.go:356] "Generic (PLEG): container finished" podID="79c978ef-76ac-44a5-9220-c574276181ae" containerID="07ca9cf20e79d2fe203ce1123fcc9abd186a0f5a28baba63896d8d5c1a192ebb" exitCode=0 Nov 03 22:39:10.003851 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.003732 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" event={"ID":"79c978ef-76ac-44a5-9220-c574276181ae","Type":"ContainerDied","Data":"07ca9cf20e79d2fe203ce1123fcc9abd186a0f5a28baba63896d8d5c1a192ebb"} Nov 03 22:39:10.096754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.085943 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:39:10.137765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.132995 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-maas-api-tls\") pod \"79c978ef-76ac-44a5-9220-c574276181ae\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " Nov 03 22:39:10.137765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.133068 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-metrics-certs\") pod \"79c978ef-76ac-44a5-9220-c574276181ae\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " Nov 03 22:39:10.137765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.133292 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c978ef-76ac-44a5-9220-c574276181ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c978ef-76ac-44a5-9220-c574276181ae-openshift-service-ca\") pod \"79c978ef-76ac-44a5-9220-c574276181ae\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " Nov 03 22:39:10.137765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.133387 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c978ef-76ac-44a5-9220-c574276181ae-kube-api-access-w57q5\" (UniqueName: \"kubernetes.io/projected/79c978ef-76ac-44a5-9220-c574276181ae-kube-api-access-w57q5\") pod \"79c978ef-76ac-44a5-9220-c574276181ae\" (UID: \"79c978ef-76ac-44a5-9220-c574276181ae\") " Nov 03 22:39:10.165760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.156159 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c978ef-76ac-44a5-9220-c574276181ae-openshift-service-ca" pod "79c978ef-76ac-44a5-9220-c574276181ae" (UID: "79c978ef-76ac-44a5-9220-c574276181ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:10.182676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.174260 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-metrics-certs" pod "79c978ef-76ac-44a5-9220-c574276181ae" (UID: "79c978ef-76ac-44a5-9220-c574276181ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:10.216134 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.211657 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-maas-api-tls" pod "79c978ef-76ac-44a5-9220-c574276181ae" (UID: "79c978ef-76ac-44a5-9220-c574276181ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:10.245569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.245410 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:10.245569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.245448 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c978ef-76ac-44a5-9220-c574276181ae-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:10.245569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.245464 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c978ef-76ac-44a5-9220-c574276181ae-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:10.358802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.330516 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c978ef-76ac-44a5-9220-c574276181ae-kube-api-access-w57q5" pod "79c978ef-76ac-44a5-9220-c574276181ae" (UID: "79c978ef-76ac-44a5-9220-c574276181ae"). InnerVolumeSpecName "kube-api-access-w57q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:10.456795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:10.375351 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57q5\" (UniqueName: \"kubernetes.io/projected/79c978ef-76ac-44a5-9220-c574276181ae-kube-api-access-w57q5\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.016772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.015655 4625 generic.go:356] "Generic (PLEG): container finished" podID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerID="8c5e5bea7a6b493710a920e5349442d17fb3ff62b4af920ae54c82dd41c18fe2" exitCode=0 Nov 03 22:39:11.016772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.015762 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" event={"ID":"32fccf19-eb27-458f-add7-b7ea20d0bc2c","Type":"ContainerDied","Data":"8c5e5bea7a6b493710a920e5349442d17fb3ff62b4af920ae54c82dd41c18fe2"} Nov 03 22:39:11.016772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.015821 4625 scope.go:122] "RemoveContainer" containerID="dd8789936a2ef062e2c8fc7e7c582335999dcfc25716a8df70dfd976b53c9318" Nov 03 22:39:11.018533 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.018507 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" event={"ID":"79c978ef-76ac-44a5-9220-c574276181ae","Type":"ContainerDied","Data":"f9c3b98ac5768b0e5d1aa4ab142b85808ddabff5c1ff70f667f78ed5005f87a2"} Nov 03 22:39:11.018533 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.018525 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj" Nov 03 22:39:11.031464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.031441 4625 scope.go:122] "RemoveContainer" containerID="07ca9cf20e79d2fe203ce1123fcc9abd186a0f5a28baba63896d8d5c1a192ebb" Nov 03 22:39:11.076683 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.076605 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj"] Nov 03 22:39:11.085818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.085792 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65b0e2-9cb7456b5-khsnj"] Nov 03 22:39:11.161072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.161035 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:39:11.191542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.191516 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-socket\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.191694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.191581 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-podinfo\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.191694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.191683 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-data\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.191822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.191800 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istiod-ca-cert\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.191881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.191834 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-socket" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:11.191966 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.191945 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-data" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:11.192031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.191961 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-token\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-token\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.192090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.192037 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-envoy\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.192090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.192063 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-kube-api-access-ld8k8\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-kube-api-access-ld8k8\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.192198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.192100 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-certs\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.192198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.192138 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-credential-socket\") pod \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\" (UID: \"32fccf19-eb27-458f-add7-b7ea20d0bc2c\") " Nov 03 22:39:11.192347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.192322 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istiod-ca-cert" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:11.192502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.192481 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-certs" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:11.192552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.192517 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-credential-socket" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:11.193114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.193091 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.193177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.193127 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.193177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.193142 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.193177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.193158 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.193177 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.193175 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.194285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.194260 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-podinfo" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:11.194285 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.194275 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-envoy" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:11.195130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.195114 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-token" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:11.195190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.195132 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-kube-api-access-ld8k8" pod "32fccf19-eb27-458f-add7-b7ea20d0bc2c" (UID: "32fccf19-eb27-458f-add7-b7ea20d0bc2c"). InnerVolumeSpecName "kube-api-access-ld8k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:11.294764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.294732 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.294764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.294761 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.294963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.294771 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8k8\" (UniqueName: \"kubernetes.io/projected/32fccf19-eb27-458f-add7-b7ea20d0bc2c-kube-api-access-ld8k8\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.294963 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.294781 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fccf19-eb27-458f-add7-b7ea20d0bc2c-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.983420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.983384 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r"] Nov 03 22:39:11.983959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:11.983651 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" containerID="cri-o://03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125" gracePeriod=30 Nov 03 22:39:12.034212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:12.034172 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" event={"ID":"32fccf19-eb27-458f-add7-b7ea20d0bc2c","Type":"ContainerDied","Data":"31db21ee29dd61ab096d461a38a253298404659a852df68656b03435aa456014"} Nov 03 22:39:12.034405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:12.034245 4625 scope.go:122] "RemoveContainer" containerID="8c5e5bea7a6b493710a920e5349442d17fb3ff62b4af920ae54c82dd41c18fe2" Nov 03 22:39:12.034405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:12.034183 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj" Nov 03 22:39:12.088603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:12.088572 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj"] Nov 03 22:39:12.093670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:12.093645 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bbfc2e-openshift-default-6c585b5767-jlcxj"] Nov 03 22:39:12.730847 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:12.730817 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" path="/var/lib/kubelet/pods/32fccf19-eb27-458f-add7-b7ea20d0bc2c/volumes" Nov 03 22:39:12.731374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:12.731360 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c978ef-76ac-44a5-9220-c574276181ae" path="/var/lib/kubelet/pods/79c978ef-76ac-44a5-9220-c574276181ae/volumes" Nov 03 22:39:16.432747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:16.428789 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6"] Nov 03 22:39:16.479239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:16.429044 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" containerID="cri-o://ca2addaadeab28f9d6cb6ae9b466f2d203c20fe6444d67a173f0aff8932b14f1" gracePeriod=30 Nov 03 22:39:17.668252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.668215 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:39:17.793841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.793747 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-kube-api-access-dgm4w\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-kube-api-access-dgm4w\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.793841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.793804 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/652d1897-361b-493e-a6ce-081d556893a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d1897-361b-493e-a6ce-081d556893a7-istio-podinfo\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.793934 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652d1897-361b-493e-a6ce-081d556893a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d1897-361b-493e-a6ce-081d556893a7-istiod-ca-cert\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.793999 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-certs\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794112 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-socket\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794218 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-istio-token\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-istio-token\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794308 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-certs" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:17.794573 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794435 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-data\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794573 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794524 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-envoy\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794573 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794315 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652d1897-361b-493e-a6ce-081d556893a7-istiod-ca-cert" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:17.794573 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794554 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-credential-socket\") pod \"652d1897-361b-493e-a6ce-081d556893a7\" (UID: \"652d1897-361b-493e-a6ce-081d556893a7\") " Nov 03 22:39:17.794843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794421 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-socket" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:17.794843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.794616 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-data" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:17.795455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.795428 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.795556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.795460 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.795556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.795475 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.795556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.795490 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d1897-361b-493e-a6ce-081d556893a7-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.795750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.795584 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-credential-socket" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:17.796440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.796395 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-istio-token" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:17.796440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.796434 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652d1897-361b-493e-a6ce-081d556893a7-istio-podinfo" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:17.796575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.796450 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-kube-api-access-dgm4w" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "kube-api-access-dgm4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:17.796765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.796744 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-envoy" pod "652d1897-361b-493e-a6ce-081d556893a7" (UID: "652d1897-361b-493e-a6ce-081d556893a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:17.902418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.900609 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.902418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.900653 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.902418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.900668 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d1897-361b-493e-a6ce-081d556893a7-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.902418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.900684 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgm4w\" (UniqueName: \"kubernetes.io/projected/652d1897-361b-493e-a6ce-081d556893a7-kube-api-access-dgm4w\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:17.902418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:17.900699 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d1897-361b-493e-a6ce-081d556893a7-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:18.215765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.215704 4625 generic.go:356] "Generic (PLEG): container finished" podID="652d1897-361b-493e-a6ce-081d556893a7" containerID="03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125" exitCode=0 Nov 03 22:39:18.215962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.215784 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" Nov 03 22:39:18.215962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.215851 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" event={"ID":"652d1897-361b-493e-a6ce-081d556893a7","Type":"ContainerDied","Data":"03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125"} Nov 03 22:39:18.215962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.215894 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r" event={"ID":"652d1897-361b-493e-a6ce-081d556893a7","Type":"ContainerDied","Data":"340178512ce4cb1505968368d61fdffadf3ebd40857f875ee7f154654f192de5"} Nov 03 22:39:18.215962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.215921 4625 scope.go:122] "RemoveContainer" containerID="03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125" Nov 03 22:39:18.231453 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.231435 4625 scope.go:122] "RemoveContainer" containerID="450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8" Nov 03 22:39:18.244154 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.244133 4625 scope.go:122] "RemoveContainer" containerID="03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125" Nov 03 22:39:18.244781 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:18.244680 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125\": container with ID starting with 03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125 not found: ID does not exist" containerID="03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125" Nov 03 22:39:18.244910 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.244795 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125"} err="failed to get container status \"03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125\": rpc error: code = NotFound desc = could not find container \"03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125\": container with ID starting with 03041d8f743a658bd5993766dd542496cf7d31d259e311aa968934879ee64125 not found: ID does not exist" Nov 03 22:39:18.244910 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.244819 4625 scope.go:122] "RemoveContainer" containerID="450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8" Nov 03 22:39:18.245248 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:18.245230 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8\": container with ID starting with 450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8 not found: ID does not exist" containerID="450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8" Nov 03 22:39:18.245324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.245252 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8"} err="failed to get container status \"450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8\": rpc error: code = NotFound desc = could not find container \"450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8\": container with ID starting with 450f9c36a39a86939f2bf53bab3e30eaee5556a669a29a9ca9db666c65b0b9a8 not found: ID does not exist" Nov 03 22:39:18.269926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.269897 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r"] Nov 03 22:39:18.281444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.281416 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-d7r4r"] Nov 03 22:39:18.838739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:18.833194 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652d1897-361b-493e-a6ce-081d556893a7" path="/var/lib/kubelet/pods/652d1897-361b-493e-a6ce-081d556893a7/volumes" Nov 03 22:39:19.685586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:19.685392 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n"] Nov 03 22:39:19.685858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:19.685826 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" containerID="cri-o://f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab" gracePeriod=30 Nov 03 22:39:22.267428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.267327 4625 generic.go:356] "Generic (PLEG): container finished" podID="af6fed8b-717b-4057-849a-69a007087715" containerID="ca2addaadeab28f9d6cb6ae9b466f2d203c20fe6444d67a173f0aff8932b14f1" exitCode=0 Nov 03 22:39:22.267428 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.267363 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" event={"ID":"af6fed8b-717b-4057-849a-69a007087715","Type":"ContainerDied","Data":"ca2addaadeab28f9d6cb6ae9b466f2d203c20fe6444d67a173f0aff8932b14f1"} Nov 03 22:39:22.458374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.458322 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:39:22.562940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.562897 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6fed8b-717b-4057-849a-69a007087715-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6fed8b-717b-4057-849a-69a007087715-istiod-ca-cert\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563091 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af6fed8b-717b-4057-849a-69a007087715-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6fed8b-717b-4057-849a-69a007087715-istio-podinfo\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563215 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563180 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-credential-socket\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563279 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563267 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-data\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563357 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-kube-api-access-pdbwj\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-kube-api-access-pdbwj\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563412 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-istio-token\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-istio-token\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563466 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-envoy\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563551 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563522 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-socket\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.563661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.563623 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-certs\") pod \"af6fed8b-717b-4057-849a-69a007087715\" (UID: \"af6fed8b-717b-4057-849a-69a007087715\") " Nov 03 22:39:22.565165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.564085 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-credential-socket" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:22.565165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.564304 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6fed8b-717b-4057-849a-69a007087715-istiod-ca-cert" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:22.565165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.564341 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-certs" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:22.565165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.564568 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-socket" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:22.565165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.564620 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.565165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.564639 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6fed8b-717b-4057-849a-69a007087715-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.565165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.564656 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.568255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.568209 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af6fed8b-717b-4057-849a-69a007087715-istio-podinfo" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:22.568586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.568551 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-istio-token" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:22.570317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.570290 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-kube-api-access-pdbwj" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "kube-api-access-pdbwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:22.570686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.570664 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-envoy" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:22.570815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.570789 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-data" pod "af6fed8b-717b-4057-849a-69a007087715" (UID: "af6fed8b-717b-4057-849a-69a007087715"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:22.666528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.666442 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6fed8b-717b-4057-849a-69a007087715-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.666528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.666480 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.666528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.666493 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdbwj\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-kube-api-access-pdbwj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.666528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.666505 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6fed8b-717b-4057-849a-69a007087715-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.666528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.666513 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:22.666528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:22.666520 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6fed8b-717b-4057-849a-69a007087715-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:23.277819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:23.277724 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" Nov 03 22:39:23.277819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:23.277765 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6" event={"ID":"af6fed8b-717b-4057-849a-69a007087715","Type":"ContainerDied","Data":"d192b2ed33617c1cb2a38315f8b52c641e046c3a4b28eb61aa2415016aa247e8"} Nov 03 22:39:23.278294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:23.277831 4625 scope.go:122] "RemoveContainer" containerID="ca2addaadeab28f9d6cb6ae9b466f2d203c20fe6444d67a173f0aff8932b14f1" Nov 03 22:39:23.340071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:23.340031 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6"] Nov 03 22:39:23.352260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:23.352227 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-9h2j6"] Nov 03 22:39:24.730467 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:24.730439 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6fed8b-717b-4057-849a-69a007087715" path="/var/lib/kubelet/pods/af6fed8b-717b-4057-849a-69a007087715/volumes" Nov 03 22:39:25.106575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.106552 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:39:25.190171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190129 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-socket\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190175 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istiod-ca-cert\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190302 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-credential-socket\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190394 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-kube-api-access-d5f5z\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-kube-api-access-d5f5z\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190413 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-socket" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:25.190604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190435 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-token\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-token\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190499 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-certs\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190553 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-envoy\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190595 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-data\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190846 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190617 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-credential-socket" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:25.190846 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190641 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-podinfo\") pod \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\" (UID: \"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8\") " Nov 03 22:39:25.190949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.190895 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istiod-ca-cert" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:25.191317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.191285 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-certs" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:25.191488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.191407 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-data" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:25.191488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.191462 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.191488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.191481 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.191663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.191495 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.191663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.191509 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.194069 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.194045 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-kube-api-access-d5f5z" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "kube-api-access-d5f5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:25.194186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.194087 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-token" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:25.194360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.194338 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-podinfo" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:25.194449 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.194396 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-envoy" pod "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" (UID: "fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:25.293008 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.292981 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.293008 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.293006 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.293008 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.293015 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.293255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.293024 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5f5z\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-kube-api-access-d5f5z\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.293255 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.293033 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:25.299243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.299211 4625 generic.go:356] "Generic (PLEG): container finished" podID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerID="f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab" exitCode=0 Nov 03 22:39:25.299383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.299257 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" event={"ID":"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8","Type":"ContainerDied","Data":"f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab"} Nov 03 22:39:25.299383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.299269 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" Nov 03 22:39:25.299383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.299314 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n" event={"ID":"fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8","Type":"ContainerDied","Data":"a79ab46746393ebf5b4ee82c901f83abaea7c5063742c0ad1251310d4e516c3a"} Nov 03 22:39:25.299383 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.299347 4625 scope.go:122] "RemoveContainer" containerID="f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab" Nov 03 22:39:25.311790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.311769 4625 scope.go:122] "RemoveContainer" containerID="739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392" Nov 03 22:39:25.325160 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.325143 4625 scope.go:122] "RemoveContainer" containerID="f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab" Nov 03 22:39:25.325609 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:25.325586 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab\": container with ID starting with f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab not found: ID does not exist" containerID="f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab" Nov 03 22:39:25.325688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.325617 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab"} err="failed to get container status \"f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab\": rpc error: code = NotFound desc = could not find container \"f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab\": container with ID starting with f8d1f84bef7b3eabbbe4b331aeabe173977c126d5fd6bde842dbac68d0756dab not found: ID does not exist" Nov 03 22:39:25.325688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.325651 4625 scope.go:122] "RemoveContainer" containerID="739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392" Nov 03 22:39:25.325927 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:25.325911 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392\": container with ID starting with 739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392 not found: ID does not exist" containerID="739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392" Nov 03 22:39:25.325985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.325931 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392"} err="failed to get container status \"739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392\": rpc error: code = NotFound desc = could not find container \"739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392\": container with ID starting with 739ea21f4ede2673b31a8db2a96618f634aeb03012b7c4ae970784bd946a8392 not found: ID does not exist" Nov 03 22:39:25.407891 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.407812 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n"] Nov 03 22:39:25.438350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:25.438313 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-65b0e2-openshift-default-56d854d69c-tdh5n"] Nov 03 22:39:26.753971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:26.751403 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" path="/var/lib/kubelet/pods/fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8/volumes" Nov 03 22:39:26.949706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:26.949672 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn"] Nov 03 22:39:26.949985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:26.949952 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" containerID="cri-o://cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3" gracePeriod=30 Nov 03 22:39:28.090556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:28.090520 4625 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:39:28.091009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:28.090582 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:39:32.202519 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:32.202476 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:39:32.259378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.259307 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:39:32.403681 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.403645 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-envoy\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.403886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.403752 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-credential-socket\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.403959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.403903 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-data\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.403959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.403937 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d50cd85-133d-447f-842c-b8debe5c5ac0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d50cd85-133d-447f-842c-b8debe5c5ac0-istiod-ca-cert\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.404075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.403963 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-certs\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.404075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404018 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-kube-api-access-pqf7f\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-kube-api-access-pqf7f\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.404075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404027 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-credential-socket" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.404246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404117 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-socket\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.404246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404156 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-token\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-token\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.404246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404227 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-podinfo\") pod \"2d50cd85-133d-447f-842c-b8debe5c5ac0\" (UID: \"2d50cd85-133d-447f-842c-b8debe5c5ac0\") " Nov 03 22:39:32.404430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404238 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-data" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.404430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404314 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d50cd85-133d-447f-842c-b8debe5c5ac0-istiod-ca-cert" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:32.404603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404581 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-socket" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.404681 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404664 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-certs" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.405014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.404991 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.405014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.405015 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.405193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.405029 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d50cd85-133d-447f-842c-b8debe5c5ac0-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.405193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.405041 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.405193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.405091 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.406544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.406518 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-envoy" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.413902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.413876 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-podinfo" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:32.414000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.413896 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-token" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:32.414000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.413937 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-kube-api-access-pqf7f" pod "2d50cd85-133d-447f-842c-b8debe5c5ac0" (UID: "2d50cd85-133d-447f-842c-b8debe5c5ac0"). InnerVolumeSpecName "kube-api-access-pqf7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:32.483917 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.483881 4625 generic.go:356] "Generic (PLEG): container finished" podID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerID="cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3" exitCode=0 Nov 03 22:39:32.484130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.483936 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" Nov 03 22:39:32.484130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.483943 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" event={"ID":"2d50cd85-133d-447f-842c-b8debe5c5ac0","Type":"ContainerDied","Data":"cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3"} Nov 03 22:39:32.484130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.483989 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn" event={"ID":"2d50cd85-133d-447f-842c-b8debe5c5ac0","Type":"ContainerDied","Data":"5ca5d3c345fc6aaac897aef9153738d2317898c3d62c27bc75c5bc3874e96036"} Nov 03 22:39:32.484130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.484011 4625 scope.go:122] "RemoveContainer" containerID="cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3" Nov 03 22:39:32.496204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.496185 4625 scope.go:122] "RemoveContainer" containerID="cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3" Nov 03 22:39:32.496532 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:32.496507 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3\": container with ID starting with cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3 not found: ID does not exist" containerID="cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3" Nov 03 22:39:32.496574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.496541 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3"} err="failed to get container status \"cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3\": rpc error: code = NotFound desc = could not find container \"cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3\": container with ID starting with cd018b19f91bb8cf015cf2879039b2b8923863af92e11cf9dede33668dc990d3 not found: ID does not exist" Nov 03 22:39:32.506700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.506675 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.506700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.506700 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.506859 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.506708 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d50cd85-133d-447f-842c-b8debe5c5ac0-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.506859 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.506736 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqf7f\" (UniqueName: \"kubernetes.io/projected/2d50cd85-133d-447f-842c-b8debe5c5ac0-kube-api-access-pqf7f\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.546003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.545935 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn"] Nov 03 22:39:32.553015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.552988 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-2gjrn"] Nov 03 22:39:32.730889 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:32.730859 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" path="/var/lib/kubelet/pods/2d50cd85-133d-447f-842c-b8debe5c5ac0/volumes" Nov 03 22:39:47.359920 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.359883 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw"] Nov 03 22:39:47.360930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.360898 4625 cpu_manager.go:414] "containerMap: removing container" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" Nov 03 22:39:47.360930 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.360925 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.360946 4625 cpu_manager.go:414] "containerMap: removing container" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.360953 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.360964 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361012 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361026 4625 cpu_manager.go:414] "containerMap: removing container" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361033 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361089 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b1db5e47-fa56-456a-9de4-bde181128810" containerName="maas-api" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361096 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1db5e47-fa56-456a-9de4-bde181128810" containerName="maas-api" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361112 4625 cpu_manager.go:414] "containerMap: removing container" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361116 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361125 4625 cpu_manager.go:414] "containerMap: removing container" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" Nov 03 22:39:47.361141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361131 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361149 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361157 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361173 4625 cpu_manager.go:414] "containerMap: removing container" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361178 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361209 4625 cpu_manager.go:414] "containerMap: removing container" podUID="79c978ef-76ac-44a5-9220-c574276181ae" containerName="maas-api" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361216 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c978ef-76ac-44a5-9220-c574276181ae" containerName="maas-api" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361411 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361423 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361430 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d50cd85-133d-447f-842c-b8debe5c5ac0" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361439 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfe290d-4fb1-4797-81d2-7f52cc7eb6c8" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361454 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1db5e47-fa56-456a-9de4-bde181128810" containerName="maas-api" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361462 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361467 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6fed8b-717b-4057-849a-69a007087715" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361473 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361480 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" Nov 03 22:39:47.361760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.361488 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c978ef-76ac-44a5-9220-c574276181ae" containerName="maas-api" Nov 03 22:39:47.366790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.366769 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.383257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.383234 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a7065-openshift-default-dockercfg-bs5ch\"" Nov 03 22:39:47.401397 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.401366 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw"] Nov 03 22:39:47.481126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481090 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481148 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxlr\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-kube-api-access-nmxlr\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481244 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-envoy\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481287 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-certs\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481307 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-credential-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481485 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-data\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481555 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-podinfo\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481658 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481639 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-token\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.481779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.481758 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istiod-ca-cert\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.598272 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-certs\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.598397 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-credential-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.598600 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-data\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.598740 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-podinfo\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.598768 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-token\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.598879 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istiod-ca-cert\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.598996 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.599208 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxlr\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-kube-api-access-nmxlr\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.602739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.599429 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-envoy\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.636798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.623107 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.636798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.623660 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-data\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.636798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.628003 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-certs\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.636798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.628689 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istiod-ca-cert\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.636798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.628987 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-credential-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.667739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.667371 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-envoy\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.696738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.683221 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-token\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.696738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.683406 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-podinfo\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.697941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.697911 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxlr\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-kube-api-access-nmxlr\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:47.983084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:47.982999 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:48.316180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.316147 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw"] Nov 03 22:39:48.316834 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:39:48.316809 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90a1d02_4cca_4c64_8d2d_485a03eac17e.slice/crio-3c81b3074a831df3eb9533fd05cf7ffdf788de02d3a3fc0902bbc12accf15eb3 WatchSource:0}: Error finding container 3c81b3074a831df3eb9533fd05cf7ffdf788de02d3a3fc0902bbc12accf15eb3: Status 404 returned error can't find the container with id 3c81b3074a831df3eb9533fd05cf7ffdf788de02d3a3fc0902bbc12accf15eb3 Nov 03 22:39:48.319644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.319610 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:39:48.319759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.319672 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:39:48.319759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.319699 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:39:48.712661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.712628 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" event={"ID":"d90a1d02-4cca-4c64-8d2d-485a03eac17e","Type":"ContainerStarted","Data":"be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d"} Nov 03 22:39:48.713180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.712691 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" event={"ID":"d90a1d02-4cca-4c64-8d2d-485a03eac17e","Type":"ContainerStarted","Data":"3c81b3074a831df3eb9533fd05cf7ffdf788de02d3a3fc0902bbc12accf15eb3"} Nov 03 22:39:48.771172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.771123 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podStartSLOduration=1.77110929 podStartE2EDuration="1.77110929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:48.767733467 +0000 UTC m=+1610.197069452" watchObservedRunningTime="2026-11-03 22:39:48.77110929 +0000 UTC m=+1610.200445354" Nov 03 22:39:48.948748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.941956 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx"] Nov 03 22:39:48.948748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:48.942215 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" podUID="ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" containerName="maas-api" containerID="cri-o://638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902" gracePeriod=30 Nov 03 22:39:49.041796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.032631 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:49.048286 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.048248 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:49.500054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.500014 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:39:49.600761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.591822 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-openshift-service-ca\") pod \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " Nov 03 22:39:49.600761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.591915 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-maas-api-tls\") pod \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " Nov 03 22:39:49.600761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.592039 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-kube-api-access-jkz5g\" (UniqueName: \"kubernetes.io/projected/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-kube-api-access-jkz5g\") pod \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " Nov 03 22:39:49.600761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.592311 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-metrics-certs\") pod \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\" (UID: \"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b\") " Nov 03 22:39:49.600761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.598279 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-openshift-service-ca" pod "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" (UID: "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:49.702066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.701972 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:49.734972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.733059 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-maas-api-tls" pod "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" (UID: "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:49.734972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.733183 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-kube-api-access-jkz5g" pod "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" (UID: "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b"). InnerVolumeSpecName "kube-api-access-jkz5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:49.734972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.733306 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-metrics-certs" pod "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" (UID: "ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:49.745303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.745273 4625 generic.go:356] "Generic (PLEG): container finished" podID="ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" containerID="638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902" exitCode=0 Nov 03 22:39:49.745946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.745925 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" Nov 03 22:39:49.746611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.746581 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" event={"ID":"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b","Type":"ContainerDied","Data":"638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902"} Nov 03 22:39:49.746731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.746667 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:49.746731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.746694 4625 scope.go:122] "RemoveContainer" containerID="638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902" Nov 03 22:39:49.746844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.746701 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx" event={"ID":"ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b","Type":"ContainerDied","Data":"2fec86c1d44e29d4bbd0974ed756402ec4c60509e62f477aa010e22fae6b2d14"} Nov 03 22:39:49.750013 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.749956 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:39:49.896872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.813465 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:49.897142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.897116 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:49.897268 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.897253 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkz5g\" (UniqueName: \"kubernetes.io/projected/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b-kube-api-access-jkz5g\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:39:49.921227 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.921204 4625 scope.go:122] "RemoveContainer" containerID="638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902" Nov 03 22:39:49.921769 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:39:49.921741 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902\": container with ID starting with 638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902 not found: ID does not exist" containerID="638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902" Nov 03 22:39:49.921956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.921930 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902"} err="failed to get container status \"638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902\": rpc error: code = NotFound desc = could not find container \"638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902\": container with ID starting with 638b12816b36fab6e013c5a89bf361a205cde550cbf9e80ad72e993ba5955902 not found: ID does not exist" Nov 03 22:39:49.953404 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.953322 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx"] Nov 03 22:39:49.991805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:49.991748 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c0223-5f5759c6bc-b9rbx"] Nov 03 22:39:50.729271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:50.729231 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" path="/var/lib/kubelet/pods/ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b/volumes" Nov 03 22:39:53.059567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.059531 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd"] Nov 03 22:39:53.060728 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.060698 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" containerName="maas-api" Nov 03 22:39:53.060823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.060734 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" containerName="maas-api" Nov 03 22:39:53.060823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.060761 4625 cpu_manager.go:414] "containerMap: removing container" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" Nov 03 22:39:53.060823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.060769 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" Nov 03 22:39:53.060823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.060822 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" Nov 03 22:39:53.060957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.060830 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cabdb81-bea7-4c0d-8d1d-8f982ffffb58" containerName="istio-proxy" Nov 03 22:39:53.061021 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.061009 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0748a6-ff5f-4aec-bc2a-bd86149d1d9b" containerName="maas-api" Nov 03 22:39:53.061053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.061028 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fccf19-eb27-458f-add7-b7ea20d0bc2c" containerName="istio-proxy" Nov 03 22:39:53.061053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.061045 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="652d1897-361b-493e-a6ce-081d556893a7" containerName="istio-proxy" Nov 03 22:39:53.065407 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.065385 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:53.081340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.081312 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd"] Nov 03 22:39:53.085653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.085605 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a7065\"" Nov 03 22:39:53.147310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.147267 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:53.147489 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.147355 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcvp\" (UniqueName: \"kubernetes.io/projected/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-kube-api-access-jfcvp\") pod \"payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:53.154800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.154772 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a7065-dockercfg-nfcqf\"" Nov 03 22:39:53.307956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.282493 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcvp\" (UniqueName: \"kubernetes.io/projected/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-kube-api-access-jfcvp\") pod \"payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:53.307956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.282857 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:53.307956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.283763 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:53.412796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.395409 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcvp\" (UniqueName: \"kubernetes.io/projected/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-kube-api-access-jfcvp\") pod \"payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:53.684312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:53.684217 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:54.015502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:54.015471 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd"] Nov 03 22:39:54.016871 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:39:54.016835 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fee21b5_5b6d_4403_b1fc_e5e8acf1feff.slice/crio-034dcaec95c5df87b2274e8143eb1ee680501fbe5866090d9d101d01e7987172 WatchSource:0}: Error finding container 034dcaec95c5df87b2274e8143eb1ee680501fbe5866090d9d101d01e7987172: Status 404 returned error can't find the container with id 034dcaec95c5df87b2274e8143eb1ee680501fbe5866090d9d101d01e7987172 Nov 03 22:39:54.796905 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:54.796869 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" event={"ID":"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff","Type":"ContainerStarted","Data":"c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623"} Nov 03 22:39:54.797403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:54.796926 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" event={"ID":"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff","Type":"ContainerStarted","Data":"034dcaec95c5df87b2274e8143eb1ee680501fbe5866090d9d101d01e7987172"} Nov 03 22:39:54.797403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:54.797007 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:39:54.846544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:54.845598 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" podStartSLOduration=1.84557601 podStartE2EDuration="1.84557601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:54.834093657 +0000 UTC m=+1616.263429673" watchObservedRunningTime="2026-11-03 22:39:54.84557601 +0000 UTC m=+1616.274912006" Nov 03 22:39:56.477471 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.477431 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf"] Nov 03 22:39:56.482019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.481996 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.520244 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.519913 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9e540d-serving-cert\"" Nov 03 22:39:56.545298 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.541309 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf"] Nov 03 22:39:56.678743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.671839 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-metrics-certs\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.678743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.671962 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795a8d6c-f9ff-48ff-bd76-8559836e1aec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.678743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.672107 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.678743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.672185 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdq4f\" (UniqueName: \"kubernetes.io/projected/795a8d6c-f9ff-48ff-bd76-8559836e1aec-kube-api-access-rdq4f\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.780789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.776019 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.780789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.776677 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdq4f\" (UniqueName: \"kubernetes.io/projected/795a8d6c-f9ff-48ff-bd76-8559836e1aec-kube-api-access-rdq4f\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.780789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.777283 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-metrics-certs\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.780789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.777401 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795a8d6c-f9ff-48ff-bd76-8559836e1aec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.797742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.789633 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795a8d6c-f9ff-48ff-bd76-8559836e1aec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.797742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.790305 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.797742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.794258 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-metrics-certs\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:56.802804 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:56.799936 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdq4f\" (UniqueName: \"kubernetes.io/projected/795a8d6c-f9ff-48ff-bd76-8559836e1aec-kube-api-access-rdq4f\") pod \"maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:57.098395 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:57.098364 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:57.433131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:57.433096 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf"] Nov 03 22:39:57.433767 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:39:57.433699 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795a8d6c_f9ff_48ff_bd76_8559836e1aec.slice/crio-90d6c39fe0d6d968310eaa0eea7087aba6e83d76e10d553646d0f209e24f93fd WatchSource:0}: Error finding container 90d6c39fe0d6d968310eaa0eea7087aba6e83d76e10d553646d0f209e24f93fd: Status 404 returned error can't find the container with id 90d6c39fe0d6d968310eaa0eea7087aba6e83d76e10d553646d0f209e24f93fd Nov 03 22:39:57.839767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:57.839668 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" event={"ID":"795a8d6c-f9ff-48ff-bd76-8559836e1aec","Type":"ContainerStarted","Data":"90d6c39fe0d6d968310eaa0eea7087aba6e83d76e10d553646d0f209e24f93fd"} Nov 03 22:39:58.085076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:58.085037 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9"] Nov 03 22:39:58.085562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:58.085306 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" containerID="cri-o://573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7" gracePeriod=30 Nov 03 22:39:58.851924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:58.851888 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" event={"ID":"795a8d6c-f9ff-48ff-bd76-8559836e1aec","Type":"ContainerStarted","Data":"c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872"} Nov 03 22:39:58.852368 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:58.852137 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:39:58.894042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:39:58.893990 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" podStartSLOduration=2.44390604 podStartE2EDuration="2.893975847s" totalImagesPullingTime="450.069807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:58.892319389 +0000 UTC m=+1620.321655381" watchObservedRunningTime="2026-11-03 22:39:58.893975847 +0000 UTC m=+1620.323311838" Nov 03 22:40:01.945737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:01.942784 4625 patch_prober.go:29] interesting pod/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:40:01.945737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:01.942881 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:40:02.188014 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:02.187979 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:40:03.849494 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.849461 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:40:03.899219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899134 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3436b774-a017-4f06-866c-a48091023a5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3436b774-a017-4f06-866c-a48091023a5b-istiod-ca-cert\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899268 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-socket\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899323 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-envoy\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899369 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3436b774-a017-4f06-866c-a48091023a5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3436b774-a017-4f06-866c-a48091023a5b-istio-podinfo\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899559 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899462 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-istio-token\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-istio-token\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899559 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899525 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-credential-socket\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899557 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3436b774-a017-4f06-866c-a48091023a5b-istiod-ca-cert" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:03.899661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899561 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-socket" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:03.899661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899632 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-data\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899794 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-certs\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.899861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899822 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-credential-socket" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:03.899861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899856 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-kube-api-access-72pz7\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-kube-api-access-72pz7\") pod \"3436b774-a017-4f06-866c-a48091023a5b\" (UID: \"3436b774-a017-4f06-866c-a48091023a5b\") " Nov 03 22:40:03.900017 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.899942 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-data" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:03.900071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.900025 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-certs" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:03.900684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.900656 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:03.900684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.900685 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:03.900940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.900701 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:03.900940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.900730 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3436b774-a017-4f06-866c-a48091023a5b-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:03.900940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.900743 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:03.902058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.902023 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3436b774-a017-4f06-866c-a48091023a5b-istio-podinfo" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:40:03.902198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.902147 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-kube-api-access-72pz7" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "kube-api-access-72pz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:03.902360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.902333 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-envoy" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:03.902776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.902749 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-istio-token" pod "3436b774-a017-4f06-866c-a48091023a5b" (UID: "3436b774-a017-4f06-866c-a48091023a5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:03.906090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.905842 4625 generic.go:356] "Generic (PLEG): container finished" podID="3436b774-a017-4f06-866c-a48091023a5b" containerID="573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7" exitCode=0 Nov 03 22:40:03.906090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.905888 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" event={"ID":"3436b774-a017-4f06-866c-a48091023a5b","Type":"ContainerDied","Data":"573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7"} Nov 03 22:40:03.906090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.905917 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" Nov 03 22:40:03.906090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.905936 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9" event={"ID":"3436b774-a017-4f06-866c-a48091023a5b","Type":"ContainerDied","Data":"ef94acf39200361c48b75aeff96dc35a0731d5a0facbc9d16c39eaac8622312a"} Nov 03 22:40:03.906090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.905963 4625 scope.go:122] "RemoveContainer" containerID="573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7" Nov 03 22:40:03.949071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.948942 4625 scope.go:122] "RemoveContainer" containerID="7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511" Nov 03 22:40:03.960898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.960876 4625 scope.go:122] "RemoveContainer" containerID="573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7" Nov 03 22:40:03.963493 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:03.962984 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7\": container with ID starting with 573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7 not found: ID does not exist" containerID="573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7" Nov 03 22:40:03.963493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.963026 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7"} err="failed to get container status \"573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7\": rpc error: code = NotFound desc = could not find container \"573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7\": container with ID starting with 573f5a84ddde711a0965bf6eada18d6de980aa7ebaeeb2ed3c1091dc8963aed7 not found: ID does not exist" Nov 03 22:40:03.963493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.963054 4625 scope.go:122] "RemoveContainer" containerID="7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511" Nov 03 22:40:03.963493 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:03.963426 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511\": container with ID starting with 7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511 not found: ID does not exist" containerID="7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511" Nov 03 22:40:03.963493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.963455 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511"} err="failed to get container status \"7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511\": rpc error: code = NotFound desc = could not find container \"7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511\": container with ID starting with 7331daa06d4f62f668c8fd1861b93ea8e2fd7ec5a2bf207fe292c9204041d511 not found: ID does not exist" Nov 03 22:40:03.988826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.988798 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9"] Nov 03 22:40:03.993101 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:03.993074 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c0223-openshift-default-6958d556b9-dwtg9"] Nov 03 22:40:04.002365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:04.002339 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3436b774-a017-4f06-866c-a48091023a5b-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:04.002365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:04.002365 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3436b774-a017-4f06-866c-a48091023a5b-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:04.002514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:04.002374 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:04.002514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:04.002382 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pz7\" (UniqueName: \"kubernetes.io/projected/3436b774-a017-4f06-866c-a48091023a5b-kube-api-access-72pz7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:04.732131 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:04.732099 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3436b774-a017-4f06-866c-a48091023a5b" path="/var/lib/kubelet/pods/3436b774-a017-4f06-866c-a48091023a5b/volumes" Nov 03 22:40:04.871013 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:04.870982 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:40:05.809028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:05.809006 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:40:16.805450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:16.805416 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7"] Nov 03 22:40:16.805979 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:16.805651 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" containerID="cri-o://940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc" gracePeriod=30 Nov 03 22:40:18.123286 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.123247 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t"] Nov 03 22:40:18.123778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.123492 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" podUID="ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" containerName="maas-api" containerID="cri-o://ccdb6c1c7e9084106bb094c10562ea6695f789017b74b438ce859ba7941b9884" gracePeriod=30 Nov 03 22:40:18.266593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.266537 4625 generic.go:356] "Generic (PLEG): container finished" podID="ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" containerID="ccdb6c1c7e9084106bb094c10562ea6695f789017b74b438ce859ba7941b9884" exitCode=0 Nov 03 22:40:18.266816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.266605 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" event={"ID":"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060","Type":"ContainerDied","Data":"ccdb6c1c7e9084106bb094c10562ea6695f789017b74b438ce859ba7941b9884"} Nov 03 22:40:18.884751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.868233 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:40:18.903740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.902213 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-maas-api-tls\") pod \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " Nov 03 22:40:18.903740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.902306 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-openshift-service-ca\") pod \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " Nov 03 22:40:18.903740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.902407 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-kube-api-access-gz4h2\" (UniqueName: \"kubernetes.io/projected/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-kube-api-access-gz4h2\") pod \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " Nov 03 22:40:18.903740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.902529 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-metrics-certs\") pod \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\" (UID: \"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060\") " Nov 03 22:40:18.928773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.927481 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-openshift-service-ca" pod "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" (UID: "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:18.933749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:18.931497 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-kube-api-access-gz4h2" pod "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" (UID: "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060"). InnerVolumeSpecName "kube-api-access-gz4h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:19.012853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.012334 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:19.012853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.012366 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4h2\" (UniqueName: \"kubernetes.io/projected/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-kube-api-access-gz4h2\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:19.014064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.014022 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-maas-api-tls" pod "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" (UID: "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:19.023393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.017335 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-metrics-certs" pod "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" (UID: "ec8ee5ab-d55e-4804-a5c8-8d39d76e1060"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:19.129740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.118382 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:19.129740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.118423 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:19.277332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.277245 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" Nov 03 22:40:19.277332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.277277 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t" event={"ID":"ec8ee5ab-d55e-4804-a5c8-8d39d76e1060","Type":"ContainerDied","Data":"6a6f3215d8009d8b928e837a13eb4a6399bded6bb72ced3682e246bfc810a816"} Nov 03 22:40:19.277572 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.277341 4625 scope.go:122] "RemoveContainer" containerID="ccdb6c1c7e9084106bb094c10562ea6695f789017b74b438ce859ba7941b9884" Nov 03 22:40:19.329947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.329914 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t"] Nov 03 22:40:19.335376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:19.335346 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b826b6-687d77975b-hdq5t"] Nov 03 22:40:20.752352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:20.752316 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" path="/var/lib/kubelet/pods/ec8ee5ab-d55e-4804-a5c8-8d39d76e1060/volumes" Nov 03 22:40:22.091813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.091785 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:40:22.187992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.187955 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-socket\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188090 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-data\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188132 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-credential-socket\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188156 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-istio-token\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-istio-token\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188179 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d419320-4e51-4e47-bb17-6a669692cf74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d419320-4e51-4e47-bb17-6a669692cf74-istiod-ca-cert\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188239 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-socket" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:22.188443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188255 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-envoy\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188283 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-certs\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188324 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-kube-api-access-5pdfv\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-kube-api-access-5pdfv\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188410 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-data" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:22.188443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188437 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-credential-socket" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:22.188849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188503 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d419320-4e51-4e47-bb17-6a669692cf74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d419320-4e51-4e47-bb17-6a669692cf74-istio-podinfo\") pod \"1d419320-4e51-4e47-bb17-6a669692cf74\" (UID: \"1d419320-4e51-4e47-bb17-6a669692cf74\") " Nov 03 22:40:22.188849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188552 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-certs" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:22.188849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.188833 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d419320-4e51-4e47-bb17-6a669692cf74-istiod-ca-cert" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:22.189275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.189252 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d419320-4e51-4e47-bb17-6a669692cf74-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.189385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.189279 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.189385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.189293 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.189385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.189305 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.189385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.189319 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.191023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.190990 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-kube-api-access-5pdfv" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "kube-api-access-5pdfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:22.191023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.191001 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d419320-4e51-4e47-bb17-6a669692cf74-istio-podinfo" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:40:22.191180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.191017 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-envoy" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:22.192306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.192279 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-istio-token" pod "1d419320-4e51-4e47-bb17-6a669692cf74" (UID: "1d419320-4e51-4e47-bb17-6a669692cf74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:22.295740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.295126 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d419320-4e51-4e47-bb17-6a669692cf74-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.295740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.295159 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.295740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.295172 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d419320-4e51-4e47-bb17-6a669692cf74-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.295740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.295186 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdfv\" (UniqueName: \"kubernetes.io/projected/1d419320-4e51-4e47-bb17-6a669692cf74-kube-api-access-5pdfv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:22.321031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.314951 4625 generic.go:356] "Generic (PLEG): container finished" podID="1d419320-4e51-4e47-bb17-6a669692cf74" containerID="940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc" exitCode=0 Nov 03 22:40:22.321031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.315068 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" event={"ID":"1d419320-4e51-4e47-bb17-6a669692cf74","Type":"ContainerDied","Data":"940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc"} Nov 03 22:40:22.321031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.315116 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" event={"ID":"1d419320-4e51-4e47-bb17-6a669692cf74","Type":"ContainerDied","Data":"88bd031c33fb560b56b60c159d68cba521a1d5071e236e5bde07ac9fee8db89e"} Nov 03 22:40:22.321031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.315144 4625 scope.go:122] "RemoveContainer" containerID="940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc" Nov 03 22:40:22.321031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.315293 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7" Nov 03 22:40:22.426193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.426163 4625 scope.go:122] "RemoveContainer" containerID="940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc" Nov 03 22:40:22.426622 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:22.426594 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc\": container with ID starting with 940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc not found: ID does not exist" containerID="940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc" Nov 03 22:40:22.427112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.426630 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc"} err="failed to get container status \"940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc\": rpc error: code = NotFound desc = could not find container \"940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc\": container with ID starting with 940cb950b94ffaae42db3e5c14a2544458c6601217290864faecc5e775a9d5fc not found: ID does not exist" Nov 03 22:40:22.472480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.472440 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7"] Nov 03 22:40:22.477510 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.477477 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-r4zr7"] Nov 03 22:40:22.729618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:22.729586 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" path="/var/lib/kubelet/pods/1d419320-4e51-4e47-bb17-6a669692cf74/volumes" Nov 03 22:40:27.274363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:27.274292 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56"] Nov 03 22:40:27.274902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:27.274581 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" containerID="cri-o://3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440" gracePeriod=30 Nov 03 22:40:28.517243 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.517194 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx"] Nov 03 22:40:28.518346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518318 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" Nov 03 22:40:28.518346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518344 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" Nov 03 22:40:28.518544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518372 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" Nov 03 22:40:28.518544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518380 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" Nov 03 22:40:28.518544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518410 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" Nov 03 22:40:28.518544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518418 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" Nov 03 22:40:28.518544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518496 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" containerName="maas-api" Nov 03 22:40:28.518544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518507 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" containerName="maas-api" Nov 03 22:40:28.518864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518757 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d419320-4e51-4e47-bb17-6a669692cf74" containerName="istio-proxy" Nov 03 22:40:28.518864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518776 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" Nov 03 22:40:28.518864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518791 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3436b774-a017-4f06-866c-a48091023a5b" containerName="istio-proxy" Nov 03 22:40:28.518864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.518805 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8ee5ab-d55e-4804-a5c8-8d39d76e1060" containerName="maas-api" Nov 03 22:40:28.524778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.524753 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.558616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.558583 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7940ea-openshift-default-dockercfg-58cvs\"" Nov 03 22:40:28.579775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.579634 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-podinfo\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.580052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.580033 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istiod-ca-cert\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.580208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.580192 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-credential-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.580363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.580349 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.580540 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.580525 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-certs\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.580672 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.580658 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-data\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.580958 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.580939 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgcg5\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-kube-api-access-sgcg5\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.591986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.591966 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-token\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.592149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.592132 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-envoy\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.592461 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.586599 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx"] Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708417 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgcg5\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-kube-api-access-sgcg5\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708475 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-token\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708513 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-envoy\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708579 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-podinfo\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708630 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istiod-ca-cert\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708684 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-credential-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708782 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708859 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-certs\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.708919 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-data\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.765758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.709479 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-data\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.778784 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.771440 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-credential-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.906380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.772045 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istiod-ca-cert\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.906606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.858505 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.927570 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.899323 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-envoy\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.927830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.906301 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-certs\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:28.973780 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:28.973670 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgcg5\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-kube-api-access-sgcg5\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:29.023578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.023055 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-podinfo\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:29.023578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.023119 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-token\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:29.098740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.093295 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp"] Nov 03 22:40:29.109749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.107754 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.137434 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.132183 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0a7065-serving-cert\"" Nov 03 22:40:29.163165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.160501 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp"] Nov 03 22:40:29.178809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.173665 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:29.204126 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.204086 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef9b2a-3bf4-403e-a701-aed993efd758-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.204277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.204206 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-metrics-certs\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.204475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.204421 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.204475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.204459 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmrs\" (UniqueName: \"kubernetes.io/projected/72ef9b2a-3bf4-403e-a701-aed993efd758-kube-api-access-jcmrs\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.306236 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.306206 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef9b2a-3bf4-403e-a701-aed993efd758-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.306436 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.306274 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-metrics-certs\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.306617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.306589 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.306797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.306642 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmrs\" (UniqueName: \"kubernetes.io/projected/72ef9b2a-3bf4-403e-a701-aed993efd758-kube-api-access-jcmrs\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.307094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.307067 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef9b2a-3bf4-403e-a701-aed993efd758-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.310369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.310155 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.313824 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.313801 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-metrics-certs\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.328887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.328856 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmrs\" (UniqueName: \"kubernetes.io/projected/72ef9b2a-3bf4-403e-a701-aed993efd758-kube-api-access-jcmrs\") pod \"maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.514798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.469740 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:29.571534 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:40:29.571487 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5bba7c_41a2_4199_9577_fa5eddbf1cde.slice/crio-d72b2f45d592d915d7e7c171a99245fada773520a6611f6379cd8dc13a55754d WatchSource:0}: Error finding container d72b2f45d592d915d7e7c171a99245fada773520a6611f6379cd8dc13a55754d: Status 404 returned error can't find the container with id d72b2f45d592d915d7e7c171a99245fada773520a6611f6379cd8dc13a55754d Nov 03 22:40:29.574365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.574340 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx"] Nov 03 22:40:29.574620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.574584 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:29.574706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.574668 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:29.574787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.574729 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:29.867968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:29.867928 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp"] Nov 03 22:40:29.870122 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:40:29.870092 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ef9b2a_3bf4_403e_a701_aed993efd758.slice/crio-5f66bc0e946ac9c240dc530bed1411543ee6492b260233f1cda13b4896e3be2e WatchSource:0}: Error finding container 5f66bc0e946ac9c240dc530bed1411543ee6492b260233f1cda13b4896e3be2e: Status 404 returned error can't find the container with id 5f66bc0e946ac9c240dc530bed1411543ee6492b260233f1cda13b4896e3be2e Nov 03 22:40:30.332332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:30.328791 4625 patch_prober.go:29] interesting pod/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:40:30.332332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:30.328859 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:40:30.482964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:30.482478 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" event={"ID":"0f5bba7c-41a2-4199-9577-fa5eddbf1cde","Type":"ContainerStarted","Data":"ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689"} Nov 03 22:40:30.482964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:30.482549 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" event={"ID":"0f5bba7c-41a2-4199-9577-fa5eddbf1cde","Type":"ContainerStarted","Data":"d72b2f45d592d915d7e7c171a99245fada773520a6611f6379cd8dc13a55754d"} Nov 03 22:40:30.484367 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:30.484342 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" event={"ID":"72ef9b2a-3bf4-403e-a701-aed993efd758","Type":"ContainerStarted","Data":"5f66bc0e946ac9c240dc530bed1411543ee6492b260233f1cda13b4896e3be2e"} Nov 03 22:40:30.528365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:30.528320 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podStartSLOduration=2.52830279 podStartE2EDuration="2.52830279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:30.524976726 +0000 UTC m=+1651.954312718" watchObservedRunningTime="2026-11-03 22:40:30.52830279 +0000 UTC m=+1651.957638784" Nov 03 22:40:31.173397 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:31.173361 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:31.180137 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:31.180110 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:31.647369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:31.647332 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" event={"ID":"72ef9b2a-3bf4-403e-a701-aed993efd758","Type":"ContainerStarted","Data":"95429a2e56433c63b1256ee982e3cf3920fbd3991382611dfd16020b54a6a955"} Nov 03 22:40:31.647556 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:31.647407 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:31.647862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:31.647841 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:31.648983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:31.648960 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:40:31.694041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:31.693988 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" podStartSLOduration=2.103841688 podStartE2EDuration="2.693973151s" totalImagesPullingTime="590.131463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:31.691669252 +0000 UTC m=+1653.121005244" watchObservedRunningTime="2026-11-03 22:40:31.693973151 +0000 UTC m=+1653.123309143" Nov 03 22:40:32.187527 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:32.187493 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:40:32.577605 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.577580 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:40:32.661064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.661024 4625 generic.go:356] "Generic (PLEG): container finished" podID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerID="3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440" exitCode=0 Nov 03 22:40:32.661264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.661090 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" Nov 03 22:40:32.661264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.661099 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" event={"ID":"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb","Type":"ContainerDied","Data":"3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440"} Nov 03 22:40:32.661264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.661189 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56" event={"ID":"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb","Type":"ContainerDied","Data":"317d5ba26ea8ea2a39d916c1ec935387f36ff01d34509d5a99407a5dfe327176"} Nov 03 22:40:32.661264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.661219 4625 scope.go:122] "RemoveContainer" containerID="3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440" Nov 03 22:40:32.676226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676200 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istiod-ca-cert\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676288 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-podinfo\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676436 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-token\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676487 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-certs\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676595 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istiod-ca-cert" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:32.676802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676642 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-envoy\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676759 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-credential-socket\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676795 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-data\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676827 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-socket\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.676988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676863 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-certs" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.676988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.676870 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-kube-api-access-wbcfd\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-kube-api-access-wbcfd\") pod \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\" (UID: \"bb3c54da-b7e6-4fdd-8a23-c424d582d8cb\") " Nov 03 22:40:32.677148 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.677094 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-data" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.677200 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.677142 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-credential-socket" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.677666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.677640 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.677793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.677670 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.677793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.677687 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.677793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.677701 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.677988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.677961 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-socket" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.678932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.678906 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-podinfo" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:40:32.679231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.679210 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-token" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:32.679503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.679472 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-kube-api-access-wbcfd" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "kube-api-access-wbcfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:32.679598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.679581 4625 scope.go:122] "RemoveContainer" containerID="98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d" Nov 03 22:40:32.680190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.680168 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-envoy" pod "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" (UID: "bb3c54da-b7e6-4fdd-8a23-c424d582d8cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.712224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.712136 4625 scope.go:122] "RemoveContainer" containerID="3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440" Nov 03 22:40:32.712543 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:32.712520 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440\": container with ID starting with 3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440 not found: ID does not exist" containerID="3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440" Nov 03 22:40:32.712632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.712553 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440"} err="failed to get container status \"3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440\": rpc error: code = NotFound desc = could not find container \"3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440\": container with ID starting with 3a240b7c83068909e4aaf7a333e1576d9520a0b7ae387a3ece182ab84baa6440 not found: ID does not exist" Nov 03 22:40:32.712632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.712570 4625 scope.go:122] "RemoveContainer" containerID="98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d" Nov 03 22:40:32.712939 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:32.712912 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d\": container with ID starting with 98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d not found: ID does not exist" containerID="98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d" Nov 03 22:40:32.713012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.712943 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d"} err="failed to get container status \"98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d\": rpc error: code = NotFound desc = could not find container \"98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d\": container with ID starting with 98d6b80bd48b346582c59dd951ee6320489f704894c0d11872ae3bca8418a96d not found: ID does not exist" Nov 03 22:40:32.779654 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.779627 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.779822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.779656 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbcfd\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-kube-api-access-wbcfd\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.779822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.779675 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.779822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.779689 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:32.779822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:32.779702 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:33.045565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:33.045481 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56"] Nov 03 22:40:33.055593 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:33.055559 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b826b6-openshift-default-785fbbd7-rwr56"] Nov 03 22:40:34.732059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.732028 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" path="/var/lib/kubelet/pods/bb3c54da-b7e6-4fdd-8a23-c424d582d8cb/volumes" Nov 03 22:40:34.801580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.801549 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl"] Nov 03 22:40:34.802287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.802272 4625 cpu_manager.go:414] "containerMap: removing container" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" Nov 03 22:40:34.802287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.802286 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" Nov 03 22:40:34.802435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.802425 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" Nov 03 22:40:34.802472 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.802442 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" Nov 03 22:40:34.808015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.807995 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:34.827036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.827011 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl"] Nov 03 22:40:34.833774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.833748 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7940ea-dockercfg-4z8pj\"" Nov 03 22:40:34.833774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.833758 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7940ea\"" Nov 03 22:40:34.971739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.971384 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e1694-9167-4b2d-9737-0283eca834e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:34.971739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:34.971456 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pl65\" (UniqueName: \"kubernetes.io/projected/ea0e1694-9167-4b2d-9737-0283eca834e2-kube-api-access-8pl65\") pod \"payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:35.075800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.075284 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e1694-9167-4b2d-9737-0283eca834e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:35.075800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.075349 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pl65\" (UniqueName: \"kubernetes.io/projected/ea0e1694-9167-4b2d-9737-0283eca834e2-kube-api-access-8pl65\") pod \"payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:35.082827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.077411 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e1694-9167-4b2d-9737-0283eca834e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:35.126896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.126864 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pl65\" (UniqueName: \"kubernetes.io/projected/ea0e1694-9167-4b2d-9737-0283eca834e2-kube-api-access-8pl65\") pod \"payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:35.423845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.423813 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:35.743511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.743487 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl"] Nov 03 22:40:35.744599 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:40:35.744575 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0e1694_9167_4b2d_9737_0283eca834e2.slice/crio-828e65e97ddd9ecac5d3d66fb77203cd661f4e728b65025f12ca766540ec0f74 WatchSource:0}: Error finding container 828e65e97ddd9ecac5d3d66fb77203cd661f4e728b65025f12ca766540ec0f74: Status 404 returned error can't find the container with id 828e65e97ddd9ecac5d3d66fb77203cd661f4e728b65025f12ca766540ec0f74 Nov 03 22:40:35.958425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.957683 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss"] Nov 03 22:40:35.958616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:35.958440 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" podUID="e3dfdae0-e574-4021-a94c-d2f62a81989a" containerName="maas-api" containerID="cri-o://e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2" gracePeriod=30 Nov 03 22:40:36.580740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.576870 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:40:36.692790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.686782 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-maas-api-tls\") pod \"e3dfdae0-e574-4021-a94c-d2f62a81989a\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " Nov 03 22:40:36.692790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.686868 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-metrics-certs\") pod \"e3dfdae0-e574-4021-a94c-d2f62a81989a\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " Nov 03 22:40:36.692790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.687107 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3dfdae0-e574-4021-a94c-d2f62a81989a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3dfdae0-e574-4021-a94c-d2f62a81989a-openshift-service-ca\") pod \"e3dfdae0-e574-4021-a94c-d2f62a81989a\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " Nov 03 22:40:36.692790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.687175 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3dfdae0-e574-4021-a94c-d2f62a81989a-kube-api-access-989fm\" (UniqueName: \"kubernetes.io/projected/e3dfdae0-e574-4021-a94c-d2f62a81989a-kube-api-access-989fm\") pod \"e3dfdae0-e574-4021-a94c-d2f62a81989a\" (UID: \"e3dfdae0-e574-4021-a94c-d2f62a81989a\") " Nov 03 22:40:36.692790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.688809 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3dfdae0-e574-4021-a94c-d2f62a81989a-openshift-service-ca" pod "e3dfdae0-e574-4021-a94c-d2f62a81989a" (UID: "e3dfdae0-e574-4021-a94c-d2f62a81989a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:36.773456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.772826 4625 generic.go:356] "Generic (PLEG): container finished" podID="e3dfdae0-e574-4021-a94c-d2f62a81989a" containerID="e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2" exitCode=0 Nov 03 22:40:36.773456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.773045 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" Nov 03 22:40:36.855092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.854806 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3dfdae0-e574-4021-a94c-d2f62a81989a-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:36.859275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.859191 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" podStartSLOduration=2.8591716050000002 podStartE2EDuration="2.859171605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:36.856153377 +0000 UTC m=+1658.285489369" watchObservedRunningTime="2026-11-03 22:40:36.859171605 +0000 UTC m=+1658.288507589" Nov 03 22:40:36.889302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.888912 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" event={"ID":"e3dfdae0-e574-4021-a94c-d2f62a81989a","Type":"ContainerDied","Data":"e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2"} Nov 03 22:40:36.889302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.888982 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss" event={"ID":"e3dfdae0-e574-4021-a94c-d2f62a81989a","Type":"ContainerDied","Data":"149f8c538c58dbb11dcd8e4f48209f53697352afae366cd2be7dfb869ed83969"} Nov 03 22:40:36.889302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.889007 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" event={"ID":"ea0e1694-9167-4b2d-9737-0283eca834e2","Type":"ContainerStarted","Data":"8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b"} Nov 03 22:40:36.889302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.889035 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" event={"ID":"ea0e1694-9167-4b2d-9737-0283eca834e2","Type":"ContainerStarted","Data":"828e65e97ddd9ecac5d3d66fb77203cd661f4e728b65025f12ca766540ec0f74"} Nov 03 22:40:36.889302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.889066 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:36.889302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.889100 4625 scope.go:122] "RemoveContainer" containerID="e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2" Nov 03 22:40:36.892084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.892054 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3dfdae0-e574-4021-a94c-d2f62a81989a-kube-api-access-989fm" pod "e3dfdae0-e574-4021-a94c-d2f62a81989a" (UID: "e3dfdae0-e574-4021-a94c-d2f62a81989a"). InnerVolumeSpecName "kube-api-access-989fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:36.896378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.896338 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-metrics-certs" pod "e3dfdae0-e574-4021-a94c-d2f62a81989a" (UID: "e3dfdae0-e574-4021-a94c-d2f62a81989a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:36.899569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.899179 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-maas-api-tls" pod "e3dfdae0-e574-4021-a94c-d2f62a81989a" (UID: "e3dfdae0-e574-4021-a94c-d2f62a81989a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:36.918923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.918893 4625 scope.go:122] "RemoveContainer" containerID="e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2" Nov 03 22:40:36.919359 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:40:36.919333 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2\": container with ID starting with e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2 not found: ID does not exist" containerID="e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2" Nov 03 22:40:36.919441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.919375 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2"} err="failed to get container status \"e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2\": rpc error: code = NotFound desc = could not find container \"e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2\": container with ID starting with e518ecffbc1df69f859c37327ee267cd0b7c73b9f5167b259065720dc95aecb2 not found: ID does not exist" Nov 03 22:40:36.956563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.956110 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-989fm\" (UniqueName: \"kubernetes.io/projected/e3dfdae0-e574-4021-a94c-d2f62a81989a-kube-api-access-989fm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:36.956563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.956142 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:36.956563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:36.956157 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3dfdae0-e574-4021-a94c-d2f62a81989a-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:40:37.213564 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:37.213473 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss"] Nov 03 22:40:37.221313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:37.221282 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5dd3b1-6b75795bcf-2hcss"] Nov 03 22:40:37.666937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:37.666904 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:40:38.551206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.551174 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp"] Nov 03 22:40:38.552468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.552438 4625 cpu_manager.go:414] "containerMap: removing container" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" Nov 03 22:40:38.552468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.552463 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3c54da-b7e6-4fdd-8a23-c424d582d8cb" containerName="istio-proxy" Nov 03 22:40:38.552648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.552494 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e3dfdae0-e574-4021-a94c-d2f62a81989a" containerName="maas-api" Nov 03 22:40:38.552648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.552503 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3dfdae0-e574-4021-a94c-d2f62a81989a" containerName="maas-api" Nov 03 22:40:38.552820 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.552693 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3dfdae0-e574-4021-a94c-d2f62a81989a" containerName="maas-api" Nov 03 22:40:38.558653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.558627 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.585207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.585172 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp"] Nov 03 22:40:38.589213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.589183 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-71a7df-openshift-default-dockercfg-q4mt7\"" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848485 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848545 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848607 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848642 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848702 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848781 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15554025-03d4-45b2-82c0-a0aa6ade4e44-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848809 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848844 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.859788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.848881 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwbz\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-kube-api-access-8rwbz\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.974739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.961591 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.974739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.973166 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.974739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.973276 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.974739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.973639 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.974739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.974017 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.977177 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.977512 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.980834 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.981233 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15554025-03d4-45b2-82c0-a0aa6ade4e44-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.981266 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.981314 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.981353 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwbz\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-kube-api-access-8rwbz\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.981751 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3dfdae0-e574-4021-a94c-d2f62a81989a" path="/var/lib/kubelet/pods/e3dfdae0-e574-4021-a94c-d2f62a81989a/volumes" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.982108 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:38.999740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:38.982484 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15554025-03d4-45b2-82c0-a0aa6ade4e44-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:39.028493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.028461 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:39.047774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.036927 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:39.074778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.064224 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwbz\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-kube-api-access-8rwbz\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:39.117792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.110138 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:39.180739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.179765 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:39.553115 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.553084 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp"] Nov 03 22:40:39.554006 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:40:39.553982 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15554025_03d4_45b2_82c0_a0aa6ade4e44.slice/crio-f91f5af05c0dd7f7379beded25f9bf67e733d6aa08a9dd0b6bc598cfaa469dbb WatchSource:0}: Error finding container f91f5af05c0dd7f7379beded25f9bf67e733d6aa08a9dd0b6bc598cfaa469dbb: Status 404 returned error can't find the container with id f91f5af05c0dd7f7379beded25f9bf67e733d6aa08a9dd0b6bc598cfaa469dbb Nov 03 22:40:39.556948 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.556913 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:39.557056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.556994 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:39.557056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.557038 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:39.968236 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.968201 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" event={"ID":"15554025-03d4-45b2-82c0-a0aa6ade4e44","Type":"ContainerStarted","Data":"b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1"} Nov 03 22:40:39.968427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:39.968263 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" event={"ID":"15554025-03d4-45b2-82c0-a0aa6ade4e44","Type":"ContainerStarted","Data":"f91f5af05c0dd7f7379beded25f9bf67e733d6aa08a9dd0b6bc598cfaa469dbb"} Nov 03 22:40:40.035538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:40.035485 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podStartSLOduration=2.035469902 podStartE2EDuration="2.035469902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:40.022593646 +0000 UTC m=+1661.451929637" watchObservedRunningTime="2026-11-03 22:40:40.035469902 +0000 UTC m=+1661.464805888" Nov 03 22:40:40.194073 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:40.192145 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:40.236447 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:40.234709 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:40.976640 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:40.976566 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:40.977710 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:40.977690 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:40:46.395217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.395185 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv"] Nov 03 22:40:46.400788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.400758 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.428233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.428202 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50b9b5-openshift-default-dockercfg-n5hxr\"" Nov 03 22:40:46.431959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.431930 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv"] Nov 03 22:40:46.493322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493289 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493332 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-envoy\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493364 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-data\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493503 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-podinfo\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493561 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvt9t\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-kube-api-access-kvt9t\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493651 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istiod-ca-cert\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493826 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-token\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493856 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-certs\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.493918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.493904 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-credential-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.595467 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.595428 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-token\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.595730 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.595692 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-certs\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.595950 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.595932 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-credential-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.596074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.596060 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.596380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.596348 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-envoy\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.596525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.596499 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-certs\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.596788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.596766 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-data\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.596961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.596894 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.597146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.597127 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-podinfo\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.597312 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.597289 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvt9t\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-kube-api-access-kvt9t\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.628736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.597475 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istiod-ca-cert\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.629368 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.607344 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-data\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.629547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.607844 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-podinfo\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.629680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.597138 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-credential-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.629792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.614699 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-envoy\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.629917 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.626988 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istiod-ca-cert\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.649987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.648445 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvt9t\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-kube-api-access-kvt9t\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.658991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.658917 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-token\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:46.746535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:46.744067 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:47.264362 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:40:47.264329 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922cd4bf_ef5a_42a1_9e00_d44e9a707d53.slice/crio-1ca2961589583f0ac38804c78dbe8d93c7cb16214c7a0d0bed4cad8e6c932861 WatchSource:0}: Error finding container 1ca2961589583f0ac38804c78dbe8d93c7cb16214c7a0d0bed4cad8e6c932861: Status 404 returned error can't find the container with id 1ca2961589583f0ac38804c78dbe8d93c7cb16214c7a0d0bed4cad8e6c932861 Nov 03 22:40:47.264999 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:47.264960 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv"] Nov 03 22:40:47.267514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:47.267483 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:47.267625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:47.267547 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:47.267625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:47.267578 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:40:47.791998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:47.791965 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:40:48.046064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:48.045959 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" event={"ID":"922cd4bf-ef5a-42a1-9e00-d44e9a707d53","Type":"ContainerStarted","Data":"356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead"} Nov 03 22:40:48.046064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:48.046031 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" event={"ID":"922cd4bf-ef5a-42a1-9e00-d44e9a707d53","Type":"ContainerStarted","Data":"1ca2961589583f0ac38804c78dbe8d93c7cb16214c7a0d0bed4cad8e6c932861"} Nov 03 22:40:48.093966 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:48.093917 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podStartSLOduration=2.093903883 podStartE2EDuration="2.093903883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:48.090241554 +0000 UTC m=+1669.519577545" watchObservedRunningTime="2026-11-03 22:40:48.093903883 +0000 UTC m=+1669.523239875" Nov 03 22:40:48.746711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:48.746682 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:48.753049 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:48.753025 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:49.058147 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:49.058070 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:40:49.059074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:40:49.059050 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:41:02.188400 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:41:02.188353 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:41:15.661746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.660056 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2"] Nov 03 22:41:15.691107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.691077 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2"] Nov 03 22:41:15.691288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.691197 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.705025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.704997 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b1ec3a\"/\"openshift-service-ca.crt\"" Nov 03 22:41:15.705186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.705048 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b1ec3a\"/\"kube-root-ca.crt\"" Nov 03 22:41:15.716563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.716534 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b1ec3a\"/\"default-dockercfg-m9jrq\"" Nov 03 22:41:15.716689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.716601 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b1ec3a\"/\"test-model-b1ec3a-kserve-self-signed-certs\"" Nov 03 22:41:15.720757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.719953 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-tmp-dir\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.720757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.720014 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183151e-5e97-4008-a434-7997f7cec128-tls-certs\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.720757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.720215 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-model-cache\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.720757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.720264 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjgq\" (UniqueName: \"kubernetes.io/projected/c183151e-5e97-4008-a434-7997f7cec128-kube-api-access-nnjgq\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.720757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.720387 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-home\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.720757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.720452 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-dshm\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.822023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.821988 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-home\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822070 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-dshm\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822334 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-tmp-dir\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822372 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183151e-5e97-4008-a434-7997f7cec128-tls-certs\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822417 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-home\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822489 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-model-cache\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822528 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjgq\" (UniqueName: \"kubernetes.io/projected/c183151e-5e97-4008-a434-7997f7cec128-kube-api-access-nnjgq\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822665 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-tmp-dir\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.824817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.822841 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-model-cache\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.825457 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.825436 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-dshm\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.825530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.825481 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183151e-5e97-4008-a434-7997f7cec128-tls-certs\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:15.846995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:15.846970 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjgq\" (UniqueName: \"kubernetes.io/projected/c183151e-5e97-4008-a434-7997f7cec128-kube-api-access-nnjgq\") pod \"test-model-b1ec3a-kserve-566b9f4fbf-85tx2\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:16.014585 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.013043 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:16.461611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.461583 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2"] Nov 03 22:41:16.462914 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:41:16.462885 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc183151e_5e97_4008_a434_7997f7cec128.slice/crio-92e136419e400381c4f76114b58886de862d17ba984658637a9f1d99439fb0c1 WatchSource:0}: Error finding container 92e136419e400381c4f76114b58886de862d17ba984658637a9f1d99439fb0c1: Status 404 returned error can't find the container with id 92e136419e400381c4f76114b58886de862d17ba984658637a9f1d99439fb0c1 Nov 03 22:41:16.672249 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.672194 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" event={"ID":"c183151e-5e97-4008-a434-7997f7cec128","Type":"ContainerStarted","Data":"a25371e603f7739605b2f9c14423258a3c47a9ef57ce5840bb348573f1baf9b0"} Nov 03 22:41:16.675271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.672820 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" event={"ID":"c183151e-5e97-4008-a434-7997f7cec128","Type":"ContainerStarted","Data":"92e136419e400381c4f76114b58886de862d17ba984658637a9f1d99439fb0c1"} Nov 03 22:41:16.677615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.677581 4625 generic.go:356] "Generic (PLEG): container finished" podID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerID="be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d" exitCode=0 Nov 03 22:41:16.677857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.677835 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" event={"ID":"d90a1d02-4cca-4c64-8d2d-485a03eac17e","Type":"ContainerDied","Data":"be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d"} Nov 03 22:41:16.678544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.678525 4625 scope.go:122] "RemoveContainer" containerID="be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d" Nov 03 22:41:16.681424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.681383 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:41:16.682239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.682211 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:41:16.682466 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.682446 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:41:16.774458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:16.774390 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podStartSLOduration=1.774370175 podStartE2EDuration="1.774370175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:16.753125845 +0000 UTC m=+1698.182461837" watchObservedRunningTime="2026-11-03 22:41:16.774370175 +0000 UTC m=+1698.203706169" Nov 03 22:41:17.017954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:17.017917 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:17.706768 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:17.697835 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" event={"ID":"d90a1d02-4cca-4c64-8d2d-485a03eac17e","Type":"ContainerStarted","Data":"56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96"} Nov 03 22:41:17.983299 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:17.983213 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:41:17.984791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:17.984762 4625 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:41:17.984910 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:17.984813 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 22:41:18.590112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:18.590062 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr"] Nov 03 22:41:18.590586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:18.590560 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" containerID="cri-o://ba89f5a79c701766c9ac245ec06d016f9890c4578f6f9b5109835a1fd73eaba3" gracePeriod=30 Nov 03 22:41:18.990795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:18.988259 4625 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:41:18.990795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:18.988353 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 22:41:19.746956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:19.746872 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:41:20.060618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:20.038761 4625 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:41:20.068350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:20.067780 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 22:41:20.985507 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:20.985471 4625 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:41:20.985792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:20.985762 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 22:41:21.999890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:21.999852 4625 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:41:22.000263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:21.999925 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 22:41:22.598344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.598298 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg"] Nov 03 22:41:22.603921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.603896 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.632306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.632264 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg"] Nov 03 22:41:22.691779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.690221 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-341c51-openshift-default-dockercfg-x6v69\"" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698098 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-certs\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698162 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-credential-socket\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698230 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-envoy\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698259 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdvt\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-kube-api-access-pkdvt\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698305 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-data\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698376 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-podinfo\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698450 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-socket\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698482 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fede61-3e56-4ddf-baa0-cfa2359c4556-istiod-ca-cert\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.705738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.698569 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-token\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.808613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.808574 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-certs\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828111 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-credential-socket\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828228 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-envoy\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828257 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdvt\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-kube-api-access-pkdvt\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828315 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-data\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828421 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-podinfo\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828514 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-socket\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828553 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fede61-3e56-4ddf-baa0-cfa2359c4556-istiod-ca-cert\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.828666 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-token\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.815203 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-certs\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.829485 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-credential-socket\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.836233 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-socket\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.837627 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fede61-3e56-4ddf-baa0-cfa2359c4556-istiod-ca-cert\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.844947 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.844108 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-data\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.868189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.868118 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-envoy\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.890992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.890961 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-token\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.925257 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.925183 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdvt\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-kube-api-access-pkdvt\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.973760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.947301 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-podinfo\") pod \"e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:22.973760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:22.959629 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:23.010271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:23.010231 4625 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:41:23.058749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:23.033185 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 22:41:24.025308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.002492 4625 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:41:24.025308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.002563 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 22:41:24.078521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.078477 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg"] Nov 03 22:41:24.111801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.111757 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:41:24.112043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.112017 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:41:24.112196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.112174 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:41:24.122162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.122133 4625 generic.go:356] "Generic (PLEG): container finished" podID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerID="ba89f5a79c701766c9ac245ec06d016f9890c4578f6f9b5109835a1fd73eaba3" exitCode=0 Nov 03 22:41:24.122274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.122138 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" event={"ID":"a812f72f-6465-4fcc-a0ed-e38e0925c91e","Type":"ContainerDied","Data":"ba89f5a79c701766c9ac245ec06d016f9890c4578f6f9b5109835a1fd73eaba3"} Nov 03 22:41:24.122337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.122311 4625 scope.go:122] "RemoveContainer" containerID="d875610069751dae6822eb42193b7d3ff5a295f65a73d1035887fbd3ce85d93a" Nov 03 22:41:24.348452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.348089 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:41:24.639018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.638983 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-kube-api-access-lwrvq\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-kube-api-access-lwrvq\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639045 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-credential-socket\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639088 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-certs\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639150 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-socket\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639201 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istiod-ca-cert\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639332 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-envoy\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639418 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-token\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-token\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639496 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-credential-socket" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:24.639582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639537 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-podinfo\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.639629 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639616 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-data\") pod \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\" (UID: \"a812f72f-6465-4fcc-a0ed-e38e0925c91e\") " Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639811 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istiod-ca-cert" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.639891 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-certs" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.640130 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-data" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.640429 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-socket" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.640767 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.640788 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.640802 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.640815 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.642297 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.640826 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.642985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.642911 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-podinfo" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:41:24.643175 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.643127 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-envoy" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:24.643397 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.643374 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-token" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:24.644764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.644735 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-kube-api-access-lwrvq" pod "a812f72f-6465-4fcc-a0ed-e38e0925c91e" (UID: "a812f72f-6465-4fcc-a0ed-e38e0925c91e"). InnerVolumeSpecName "kube-api-access-lwrvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:24.749163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.749112 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwrvq\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-kube-api-access-lwrvq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.749363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.749330 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.749363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.749355 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:24.749582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:24.749368 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812f72f-6465-4fcc-a0ed-e38e0925c91e-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:41:25.112087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.112052 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:41:25.130739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.121992 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:41:25.204394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.204350 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" event={"ID":"a812f72f-6465-4fcc-a0ed-e38e0925c91e","Type":"ContainerDied","Data":"e60c3926488a4cab842c0fb4a90d10dcfc790682d0e23c9f69f4e5eab0c137d0"} Nov 03 22:41:25.204803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.204779 4625 scope.go:122] "RemoveContainer" containerID="ba89f5a79c701766c9ac245ec06d016f9890c4578f6f9b5109835a1fd73eaba3" Nov 03 22:41:25.205162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.205148 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" Nov 03 22:41:25.217620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.217576 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" event={"ID":"22fede61-3e56-4ddf-baa0-cfa2359c4556","Type":"ContainerStarted","Data":"e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558"} Nov 03 22:41:25.217926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.217835 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" event={"ID":"22fede61-3e56-4ddf-baa0-cfa2359c4556","Type":"ContainerStarted","Data":"abb612519734e91e29aeb3dcba1703d7c5989e77c6f8844a0ca1b10adf8dc5bf"} Nov 03 22:41:25.284493 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.282249 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr"] Nov 03 22:41:25.295796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.295173 4625 patch_prober.go:29] interesting pod/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:41:25.295796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.295256 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:25.297921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.297892 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-22edc1-openshift-default-b9f7f8cb5-qvnsr"] Nov 03 22:41:25.333240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:25.333053 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podStartSLOduration=3.333038367 podStartE2EDuration="3.333038367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:25.330111198 +0000 UTC m=+1706.759447190" watchObservedRunningTime="2026-11-03 22:41:25.333038367 +0000 UTC m=+1706.762374359" Nov 03 22:41:26.009824 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.008047 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:26.034744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.033913 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:26.279659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.279562 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:26.284514 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.284486 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:26.295195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.295166 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:41:26.300684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.300660 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:26.370148 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.370118 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:26.729579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:26.729544 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" path="/var/lib/kubelet/pods/a812f72f-6465-4fcc-a0ed-e38e0925c91e/volumes" Nov 03 22:41:31.332267 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.332236 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt"] Nov 03 22:41:31.333418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.333389 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" Nov 03 22:41:31.333801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.333778 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" Nov 03 22:41:31.334247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.334223 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" Nov 03 22:41:31.334362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.334261 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" Nov 03 22:41:31.340531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.340510 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.366414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.366378 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt"] Nov 03 22:41:31.373045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.372977 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-50b9b5-serving-cert\"" Nov 03 22:41:31.449699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.449668 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-metrics-certs\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.449925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.449858 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95233eed-0921-4a71-9cf8-3b48f7958b4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.449992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.449938 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.449992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.449984 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kn8d\" (UniqueName: \"kubernetes.io/projected/95233eed-0921-4a71-9cf8-3b48f7958b4d-kube-api-access-9kn8d\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.557738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.553800 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-metrics-certs\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.557738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.553906 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95233eed-0921-4a71-9cf8-3b48f7958b4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.557738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.553944 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.557738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.553991 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kn8d\" (UniqueName: \"kubernetes.io/projected/95233eed-0921-4a71-9cf8-3b48f7958b4d-kube-api-access-9kn8d\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.573738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.566524 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95233eed-0921-4a71-9cf8-3b48f7958b4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.573738 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:41:31.566663 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-serving-cert: secret "maas-api-e2e-inf-a-50b9b5-serving-cert" not found Nov 03 22:41:31.573738 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:41:31.566758 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls podName:95233eed-0921-4a71-9cf8-3b48f7958b4d nodeName:}" failed. No retries permitted until 2026-11-03 22:41:32.066737129 +0000 UTC m=+1713.496073105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls") pod "maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" (UID: "95233eed-0921-4a71-9cf8-3b48f7958b4d") : secret "maas-api-e2e-inf-a-50b9b5-serving-cert" not found Nov 03 22:41:31.573738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.570322 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-metrics-certs\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:31.630690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:31.626567 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kn8d\" (UniqueName: \"kubernetes.io/projected/95233eed-0921-4a71-9cf8-3b48f7958b4d-kube-api-access-9kn8d\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:32.165553 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:32.165523 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:32.168445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:32.168418 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:32.195726 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:41:32.195668 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:41:32.280734 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:32.278042 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:32.808512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:32.808481 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt"] Nov 03 22:41:32.809657 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:41:32.809629 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95233eed_0921_4a71_9cf8_3b48f7958b4d.slice/crio-b5ddcf60c31065e4514eb18976efce94c0517f4de1c4aa3924b3e3877e18c192 WatchSource:0}: Error finding container b5ddcf60c31065e4514eb18976efce94c0517f4de1c4aa3924b3e3877e18c192: Status 404 returned error can't find the container with id b5ddcf60c31065e4514eb18976efce94c0517f4de1c4aa3924b3e3877e18c192 Nov 03 22:41:32.811157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:32.811138 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:41:33.359868 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:33.359830 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" event={"ID":"95233eed-0921-4a71-9cf8-3b48f7958b4d","Type":"ContainerStarted","Data":"23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307"} Nov 03 22:41:33.360066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:33.359893 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" event={"ID":"95233eed-0921-4a71-9cf8-3b48f7958b4d","Type":"ContainerStarted","Data":"b5ddcf60c31065e4514eb18976efce94c0517f4de1c4aa3924b3e3877e18c192"} Nov 03 22:41:33.360066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:33.359947 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:33.403709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:33.403607 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" podStartSLOduration=2.088854155 podStartE2EDuration="2.403591769s" totalImagesPullingTime="314.737614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:33.400597026 +0000 UTC m=+1714.829933016" watchObservedRunningTime="2026-11-03 22:41:33.403591769 +0000 UTC m=+1714.832927761" Nov 03 22:41:39.375005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:39.374973 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:41:54.844749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:54.834392 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:41:54.961061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:54.960994 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.225933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:55.220849 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.306892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:55.306839 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.408076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:55.408018 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.873906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:55.862943 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.897747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:55.890303 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:41:55.979841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:55.979796 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.997923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:55.980533 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:41:56.249800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.224559 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:56.284739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.250408 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:41:56.284739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.268771 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq"] Nov 03 22:41:56.284739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.269050 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" containerID="cri-o://508f43fe2c469e18979c12ccf7f6c3e0666ac5a3d27bc5caaa82a06038f4bb06" gracePeriod=30 Nov 03 22:41:56.334743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.313931 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:56.334743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.316856 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:41:56.602985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.602943 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:41:56.901945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.865880 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf"] Nov 03 22:41:56.901945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.866286 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:41:56.901945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.868624 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" containerID="cri-o://d2dbadc7964719a21c09e3bcfc6fe2c033663ac7436116997406ea3662d5b9c6" gracePeriod=30 Nov 03 22:41:56.905765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:56.902819 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:57.014114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:57.012868 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:57.086782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:57.059959 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/health\": context deadline exceeded" Nov 03 22:41:57.332801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:57.328007 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:57.530951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:57.530897 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:57.831747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:57.822810 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:57.931891 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:57.931847 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:58.047855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:58.019530 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:58.198743 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:41:58.197815 4625 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.101s" Nov 03 22:41:58.358806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:58.342823 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:58.534525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:58.534420 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:58.848794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:58.772856 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:58.849353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:58.849299 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:58.939759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:58.937888 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:59.020821 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:59.019837 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:41:59.354146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:59.354107 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:59.539737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:59.538800 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:59.862345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:59.853814 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:41:59.986918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:41:59.986863 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:00.022855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:00.022807 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:00.436871 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:00.436821 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:00.604737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:00.599627 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded" Nov 03 22:42:00.855171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:00.855131 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:00.996740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:00.996037 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:42:01.027375 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:01.027335 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:42:01.391324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:01.390146 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:42:01.467735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:01.441331 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:01.740813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:01.712798 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:42:01.827879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:01.805858 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded" Nov 03 22:42:01.951738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:01.948615 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:02.019365 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:02.019265 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:02.032352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:02.032218 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:42:02.519955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:02.419756 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:42:02.613001 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:42:02.611831 4625 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694ccfdd_7723_4d33_b5c8_98585ffcb013.slice/crio-49798e2634606ef69862b60e8ea6bf79cfd5f6fa42959926035a16551d892a7b/memory.min": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694ccfdd_7723_4d33_b5c8_98585ffcb013.slice/crio-49798e2634606ef69862b60e8ea6bf79cfd5f6fa42959926035a16551d892a7b/memory.min: no such device Nov 03 22:42:02.731739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:02.672731 4625 generic.go:356] "Generic (PLEG): container finished" podID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerID="508f43fe2c469e18979c12ccf7f6c3e0666ac5a3d27bc5caaa82a06038f4bb06" exitCode=0 Nov 03 22:42:02.803366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:02.799644 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" event={"ID":"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d","Type":"ContainerDied","Data":"508f43fe2c469e18979c12ccf7f6c3e0666ac5a3d27bc5caaa82a06038f4bb06"} Nov 03 22:42:02.895152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:02.855853 4625 scope.go:122] "RemoveContainer" containerID="1376f0546248cd71ed2616cf361be767a8a0e40766b5c3a473e51b6b20a6d104" Nov 03 22:42:02.895152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:02.856135 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:03.228999 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:42:03.228906 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:42:03.275741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:03.251412 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:42:03.405741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:03.405249 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:03.860970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:03.860893 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:03.930810 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:42:03.921411 4625 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.083s" Nov 03 22:42:03.991464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:03.991430 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w"] Nov 03 22:42:03.991758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:03.951784 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:04.020739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.002602 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" containerID="cri-o://9444f2fc246c65a0468fc8ba88b5216c44c68a8906d07994860c6dce38994fd0" gracePeriod=30 Nov 03 22:42:04.257204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.257163 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:42:04.394838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.294937 4625 generic.go:356] "Generic (PLEG): container finished" podID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerID="d2dbadc7964719a21c09e3bcfc6fe2c033663ac7436116997406ea3662d5b9c6" exitCode=0 Nov 03 22:42:04.394838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.294965 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" event={"ID":"694ccfdd-7723-4d33-b5c8-98585ffcb013","Type":"ContainerDied","Data":"d2dbadc7964719a21c09e3bcfc6fe2c033663ac7436116997406ea3662d5b9c6"} Nov 03 22:42:04.394838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.392505 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:42:04.490552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.479742 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:04.499745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.498811 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:42:04.834467 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.834353 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 03 22:42:04.875885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.861899 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:04.926047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.924999 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:04.985079 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:42:04.970918 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694ccfdd_7723_4d33_b5c8_98585ffcb013.slice/crio-d2dbadc7964719a21c09e3bcfc6fe2c033663ac7436116997406ea3662d5b9c6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694ccfdd_7723_4d33_b5c8_98585ffcb013.slice/crio-conmon-d2dbadc7964719a21c09e3bcfc6fe2c033663ac7436116997406ea3662d5b9c6.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:42:04.985079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.984371 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:42:04.997244 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.996777 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:42:04.997244 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:04.996854 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:42:05.002789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.002765 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:42:05.005908 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.004232 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:42:05.063866 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.063699 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-certs\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.063866 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.063776 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-podinfo\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.064448 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.064404 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-certs" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.064946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.064640 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-socket\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.064946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.064738 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-data\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.064946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.064771 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-credential-socket\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.064946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.064849 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-credential-socket\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.064946 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.064884 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-data\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.065253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.064975 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-socket" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.065253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065129 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-credential-socket" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.065374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065256 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-data" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065453 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-credential-socket" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065537 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-podinfo\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065579 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-token\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-token\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065653 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-envoy\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065684 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-kube-api-access-pdls2\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-kube-api-access-pdls2\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065740 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694ccfdd-7723-4d33-b5c8-98585ffcb013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694ccfdd-7723-4d33-b5c8-98585ffcb013-istiod-ca-cert\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065777 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-certs\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065786 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-data" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065838 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-kube-api-access-4dnpw\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-kube-api-access-4dnpw\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065870 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-socket\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065927 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istiod-ca-cert\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.065984 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-token\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-token\") pod \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\" (UID: \"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.066015 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-envoy\") pod \"694ccfdd-7723-4d33-b5c8-98585ffcb013\" (UID: \"694ccfdd-7723-4d33-b5c8-98585ffcb013\") " Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.066804 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.066826 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.066841 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.066854 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.066867 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.067529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.066880 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.069742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.069428 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-socket" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.069742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.069325 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-envoy" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.070778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.070062 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-certs" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.070778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.070708 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694ccfdd-7723-4d33-b5c8-98585ffcb013-istiod-ca-cert" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:05.071709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.071433 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istiod-ca-cert" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:05.072728 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.072669 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-podinfo" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:05.076456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.076416 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-podinfo" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:05.076618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.076601 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-kube-api-access-pdls2" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "kube-api-access-pdls2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:05.076848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.076749 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-token" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:05.077269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.077240 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-envoy" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.077933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.077902 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-token" pod "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" (UID: "544947a5-5b4d-4d74-a1f0-2d8c7b1a514d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:05.079070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.079044 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-kube-api-access-4dnpw" pod "694ccfdd-7723-4d33-b5c8-98585ffcb013" (UID: "694ccfdd-7723-4d33-b5c8-98585ffcb013"). InnerVolumeSpecName "kube-api-access-4dnpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:05.105231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.105181 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:42:05.105386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.105260 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170135 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170169 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdls2\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-kube-api-access-pdls2\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170184 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694ccfdd-7723-4d33-b5c8-98585ffcb013-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170198 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170214 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnpw\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-kube-api-access-4dnpw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170226 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170240 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170254 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170266 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170276 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170288 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.171945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.170300 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694ccfdd-7723-4d33-b5c8-98585ffcb013-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:05.677176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.677141 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" event={"ID":"544947a5-5b4d-4d74-a1f0-2d8c7b1a514d","Type":"ContainerDied","Data":"0f3d8be0b9c758f8816abdb62a491ba521071fe580e6df8a37ea2603ba4afe4d"} Nov 03 22:42:05.683199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.683012 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" event={"ID":"694ccfdd-7723-4d33-b5c8-98585ffcb013","Type":"ContainerDied","Data":"49798e2634606ef69862b60e8ea6bf79cfd5f6fa42959926035a16551d892a7b"} Nov 03 22:42:05.683199 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.682381 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf" Nov 03 22:42:05.685703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.677920 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq" Nov 03 22:42:05.688643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.688437 4625 scope.go:122] "RemoveContainer" containerID="508f43fe2c469e18979c12ccf7f6c3e0666ac5a3d27bc5caaa82a06038f4bb06" Nov 03 22:42:05.723321 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.722913 4625 scope.go:122] "RemoveContainer" containerID="d2dbadc7964719a21c09e3bcfc6fe2c033663ac7436116997406ea3662d5b9c6" Nov 03 22:42:05.909994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.909918 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf"] Nov 03 22:42:05.925094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.925054 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-7kgzf"] Nov 03 22:42:05.937525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.937434 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq"] Nov 03 22:42:05.955789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.948060 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-tdqsq"] Nov 03 22:42:05.985242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.985193 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7"] Nov 03 22:42:05.989127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989099 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" Nov 03 22:42:05.989127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989126 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a812f72f-6465-4fcc-a0ed-e38e0925c91e" containerName="istio-proxy" Nov 03 22:42:05.989315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989199 4625 cpu_manager.go:414] "containerMap: removing container" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" Nov 03 22:42:05.989315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989206 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" Nov 03 22:42:05.989315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989234 4625 cpu_manager.go:414] "containerMap: removing container" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" Nov 03 22:42:05.989315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989241 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" Nov 03 22:42:05.989315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989260 4625 cpu_manager.go:414] "containerMap: removing container" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" Nov 03 22:42:05.989315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989295 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" Nov 03 22:42:05.989504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989487 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" Nov 03 22:42:05.989538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989504 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" containerName="istio-proxy" Nov 03 22:42:05.989538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.989516 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" containerName="istio-proxy" Nov 03 22:42:06.008575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:05.995583 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.008575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.007815 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7"] Nov 03 22:42:06.011429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.011404 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bb1fa5-serving-cert\"" Nov 03 22:42:06.155417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.154857 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.155417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.155045 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-metrics-certs\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.155417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.155153 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxb4\" (UniqueName: \"kubernetes.io/projected/77d251f7-a96d-410e-9c90-507c492aee37-kube-api-access-scxb4\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.155417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.155192 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d251f7-a96d-410e-9c90-507c492aee37-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.263968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.257418 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-metrics-certs\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.263968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.257556 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxb4\" (UniqueName: \"kubernetes.io/projected/77d251f7-a96d-410e-9c90-507c492aee37-kube-api-access-scxb4\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.263968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.257597 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d251f7-a96d-410e-9c90-507c492aee37-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.263968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.257742 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.263968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.258647 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d251f7-a96d-410e-9c90-507c492aee37-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.265410 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.265351 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.275381 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.275347 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-metrics-certs\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.283249 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.283217 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxb4\" (UniqueName: \"kubernetes.io/projected/77d251f7-a96d-410e-9c90-507c492aee37-kube-api-access-scxb4\") pod \"maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.322292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.322259 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:06.753363 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.753315 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544947a5-5b4d-4d74-a1f0-2d8c7b1a514d" path="/var/lib/kubelet/pods/544947a5-5b4d-4d74-a1f0-2d8c7b1a514d/volumes" Nov 03 22:42:06.754281 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.754258 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694ccfdd-7723-4d33-b5c8-98585ffcb013" path="/var/lib/kubelet/pods/694ccfdd-7723-4d33-b5c8-98585ffcb013/volumes" Nov 03 22:42:06.866499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:06.866406 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7"] Nov 03 22:42:07.090266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:07.090210 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:42:07.090430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:07.090273 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:42:07.711739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:07.710378 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" event={"ID":"77d251f7-a96d-410e-9c90-507c492aee37","Type":"ContainerStarted","Data":"342c59d13806de580d4d80d24bf559191a7bdf5b35333ab7a15c7f74a477c16c"} Nov 03 22:42:07.711739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:07.710452 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" event={"ID":"77d251f7-a96d-410e-9c90-507c492aee37","Type":"ContainerStarted","Data":"df263a71a4e0fc1a21d30fc9644de5f7307347c072924a813d39603479c4d3d8"} Nov 03 22:42:07.711739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:07.710817 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:07.765547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:07.764096 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" podStartSLOduration=2.320758402 podStartE2EDuration="2.764082489s" totalImagesPullingTime="443.324087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:07.757099009 +0000 UTC m=+1749.186435046" watchObservedRunningTime="2026-11-03 22:42:07.764082489 +0000 UTC m=+1749.193418495" Nov 03 22:42:08.998678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:08.998639 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv"] Nov 03 22:42:09.028853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.028602 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.085871 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.084551 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv"] Nov 03 22:42:09.093015 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.092979 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093084 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddzbp\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-kube-api-access-ddzbp\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093147 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093253 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093329 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093453 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093382 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093453 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093420 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093474 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.093663 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.093643 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195128 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195090 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195202 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddzbp\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-kube-api-access-ddzbp\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195260 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195328 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195307 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195369 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195415 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195454 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195549 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195510 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195655 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195791 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195745 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.195902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.195787 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.196422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.196122 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.196422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.196299 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.196422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.196316 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.198460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.198431 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.198914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.198894 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.217783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.217758 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.217918 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.217794 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddzbp\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-kube-api-access-ddzbp\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.457932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.457887 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:09.734924 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.734829 4625 generic.go:356] "Generic (PLEG): container finished" podID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerID="9444f2fc246c65a0468fc8ba88b5216c44c68a8906d07994860c6dce38994fd0" exitCode=0 Nov 03 22:42:09.735098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.735012 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" event={"ID":"c5b6ba1a-9a96-4ae3-af4a-413cab63c841","Type":"ContainerDied","Data":"9444f2fc246c65a0468fc8ba88b5216c44c68a8906d07994860c6dce38994fd0"} Nov 03 22:42:09.871016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.870198 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:09.871016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.870292 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:09.871016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.870336 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:09.875811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.875779 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv"] Nov 03 22:42:09.889395 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:09.889367 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:42:10.036442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.035616 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-podinfo\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.036442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.035680 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-envoy\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.036442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.035902 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-data\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.036442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.036023 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-certs\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.038465 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.037164 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-token\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.038465 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.037243 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istiod-ca-cert\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.038465 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.037283 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-socket\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.038465 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.037371 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-kube-api-access-59lfl\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-kube-api-access-59lfl\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.038465 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.037406 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-credential-socket\") pod \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\" (UID: \"c5b6ba1a-9a96-4ae3-af4a-413cab63c841\") " Nov 03 22:42:10.039412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.039106 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-credential-socket" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:10.040627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.040340 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-socket" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:10.040749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.040684 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istiod-ca-cert" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:10.040957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.040936 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-data" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:10.041153 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.041134 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-certs" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:10.045265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.045238 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-token" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:10.049366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.049320 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-kube-api-access-59lfl" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "kube-api-access-59lfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:10.094487 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.094426 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-podinfo" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:10.100939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.100899 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-envoy" pod "c5b6ba1a-9a96-4ae3-af4a-413cab63c841" (UID: "c5b6ba1a-9a96-4ae3-af4a-413cab63c841"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:10.146014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.145969 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146011 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146026 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146040 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146053 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146065 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59lfl\" (UniqueName: \"kubernetes.io/projected/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-kube-api-access-59lfl\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146079 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146091 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.146344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.146104 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ba1a-9a96-4ae3-af4a-413cab63c841-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:42:10.956755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.948983 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" Nov 03 22:42:10.971508 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.971472 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w" event={"ID":"c5b6ba1a-9a96-4ae3-af4a-413cab63c841","Type":"ContainerDied","Data":"7f39e366b143cfe7f76b7eb1af554ff4362470d0ba428dba632fdde98b7318b7"} Nov 03 22:42:10.971785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.971762 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" event={"ID":"178565ed-ea9b-4f39-8cfc-c6b0415bacf2","Type":"ContainerStarted","Data":"6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205"} Nov 03 22:42:10.971923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.971910 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" event={"ID":"178565ed-ea9b-4f39-8cfc-c6b0415bacf2","Type":"ContainerStarted","Data":"2a7a5371ddaf99a95dc7756b14d5842c5a1703303f01e06dff8fec41a2d454f8"} Nov 03 22:42:10.972032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:10.972021 4625 scope.go:122] "RemoveContainer" containerID="9444f2fc246c65a0468fc8ba88b5216c44c68a8906d07994860c6dce38994fd0" Nov 03 22:42:11.035962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.035899 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podStartSLOduration=3.035879021 podStartE2EDuration="3.035879021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:11.029504048 +0000 UTC m=+1752.458840041" watchObservedRunningTime="2026-11-03 22:42:11.035879021 +0000 UTC m=+1752.465215015" Nov 03 22:42:11.071616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.071577 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w"] Nov 03 22:42:11.081058 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.081014 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-sjh6w"] Nov 03 22:42:11.190801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.185383 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg"] Nov 03 22:42:11.190801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.186676 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" Nov 03 22:42:11.190801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.186701 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" Nov 03 22:42:11.190801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.186923 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" containerName="istio-proxy" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.205618 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.207781 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg"] Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.231678 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-341c51-serving-cert\"" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.313797 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.313845 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27xv\" (UniqueName: \"kubernetes.io/projected/6b316235-3895-40b3-b3dd-623281a7e2dc-kube-api-access-l27xv\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.313888 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-metrics-certs\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.314046 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b316235-3895-40b3-b3dd-623281a7e2dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.422985 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.423036 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27xv\" (UniqueName: \"kubernetes.io/projected/6b316235-3895-40b3-b3dd-623281a7e2dc-kube-api-access-l27xv\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.423078 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-metrics-certs\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.423252 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b316235-3895-40b3-b3dd-623281a7e2dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.424329 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b316235-3895-40b3-b3dd-623281a7e2dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:42:11.424454 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-serving-cert: secret "maas-api-e2e-inf-a-341c51-serving-cert" not found Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:42:11.424512 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls podName:6b316235-3895-40b3-b3dd-623281a7e2dc nodeName:}" failed. No retries permitted until 2026-11-03 22:42:11.924494069 +0000 UTC m=+1753.353830052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls") pod "maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" (UID: "6b316235-3895-40b3-b3dd-623281a7e2dc") : secret "maas-api-e2e-inf-a-341c51-serving-cert" not found Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.453345 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-metrics-certs\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.463449 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.466562 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27xv\" (UniqueName: \"kubernetes.io/projected/6b316235-3895-40b3-b3dd-623281a7e2dc-kube-api-access-l27xv\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.471142 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:11.926503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.471192 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:11.945983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.930830 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:12.023569 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:11.961853 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:12.030763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:12.026840 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:42:12.030763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:12.026897 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:12.155735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:12.151971 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:12.459840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:12.459712 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:12.460018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:12.459849 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:12.810961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:12.805295 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b6ba1a-9a96-4ae3-af4a-413cab63c841" path="/var/lib/kubelet/pods/c5b6ba1a-9a96-4ae3-af4a-413cab63c841/volumes" Nov 03 22:42:12.810961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:12.805657 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg"] Nov 03 22:42:13.023543 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.023509 4625 generic.go:356] "Generic (PLEG): container finished" podID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerID="b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1" exitCode=0 Nov 03 22:42:13.023742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.023582 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" event={"ID":"15554025-03d4-45b2-82c0-a0aa6ade4e44","Type":"ContainerDied","Data":"b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1"} Nov 03 22:42:13.024166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.024138 4625 scope.go:122] "RemoveContainer" containerID="b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1" Nov 03 22:42:13.025764 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.025702 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" event={"ID":"6b316235-3895-40b3-b3dd-623281a7e2dc","Type":"ContainerStarted","Data":"0f7e448ebde1ee14d897a6ca9735ebeb800549b031e196eff6c896b4ae297b73"} Nov 03 22:42:13.026258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.026230 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.026374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.026300 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.026374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.026336 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.027531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.027509 4625 generic.go:356] "Generic (PLEG): container finished" podID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerID="356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead" exitCode=0 Nov 03 22:42:13.027621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.027583 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" event={"ID":"922cd4bf-ef5a-42a1-9e00-d44e9a707d53","Type":"ContainerDied","Data":"356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead"} Nov 03 22:42:13.028074 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.028056 4625 scope.go:122] "RemoveContainer" containerID="356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead" Nov 03 22:42:13.059781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.058780 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.059781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.058887 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.059781 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.058935 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.069554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.067495 4625 generic.go:356] "Generic (PLEG): container finished" podID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerID="ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689" exitCode=0 Nov 03 22:42:13.069554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.067679 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" event={"ID":"0f5bba7c-41a2-4199-9577-fa5eddbf1cde","Type":"ContainerDied","Data":"ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689"} Nov 03 22:42:13.069554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.068269 4625 scope.go:122] "RemoveContainer" containerID="ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689" Nov 03 22:42:13.070877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.070843 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.070976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.070933 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.071016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.070980 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:13.498742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.482534 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:13.498742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.496077 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:13.787159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:13.785898 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:42:14.214740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.212391 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" event={"ID":"0f5bba7c-41a2-4199-9577-fa5eddbf1cde","Type":"ContainerStarted","Data":"ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf"} Nov 03 22:42:14.216857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.216824 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" event={"ID":"15554025-03d4-45b2-82c0-a0aa6ade4e44","Type":"ContainerStarted","Data":"9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297"} Nov 03 22:42:14.293193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.293156 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" event={"ID":"6b316235-3895-40b3-b3dd-623281a7e2dc","Type":"ContainerStarted","Data":"e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539"} Nov 03 22:42:14.298741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.293876 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:14.372624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.372196 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" event={"ID":"922cd4bf-ef5a-42a1-9e00-d44e9a707d53","Type":"ContainerStarted","Data":"9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119"} Nov 03 22:42:14.449741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.444457 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" podStartSLOduration=2.9238443419999998 podStartE2EDuration="3.444434903s" totalImagesPullingTime="520.590561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:14.438042298 +0000 UTC m=+1755.867378292" watchObservedRunningTime="2026-11-03 22:42:14.444434903 +0000 UTC m=+1755.873770895" Nov 03 22:42:14.477794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.476944 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:14.477794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.477001 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:14.759796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.759443 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:14.759796 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.759511 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:14.822009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:14.821245 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:42:15.188744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.183437 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:42:15.188744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.183590 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:15.188744 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.183634 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:15.189171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.189141 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:42:15.204890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.204853 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:15.205159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.205126 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:15.537845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.533845 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:15.537845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.533911 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:15.626742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.604276 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v"] Nov 03 22:42:15.783740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.760482 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.834020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.781384 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-model-cache\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.838984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.838556 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:15.838984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.838612 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:15.839799 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.839675 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tls-certs\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.852145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.852100 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt9mw\" (UniqueName: \"kubernetes.io/projected/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-kube-api-access-wt9mw\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.853765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.853738 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-dshm\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.858812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.854589 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-home\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.875774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.872022 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e540d\"/\"test-model-9e540d-kserve-self-signed-certs\"" Nov 03 22:42:15.875774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.872123 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e540d\"/\"kube-root-ca.crt\"" Nov 03 22:42:15.875774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.872234 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e540d\"/\"default-dockercfg-8vj6r\"" Nov 03 22:42:15.875774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.872290 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e540d\"/\"openshift-service-ca.crt\"" Nov 03 22:42:15.875774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.873376 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tmp-dir\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.897123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.897032 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v"] Nov 03 22:42:15.995743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.979661 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tmp-dir\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.995743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.979794 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-model-cache\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.995743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.979894 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tls-certs\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.995743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.979946 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt9mw\" (UniqueName: \"kubernetes.io/projected/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-kube-api-access-wt9mw\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.995743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.979987 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-dshm\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.995743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.980113 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-home\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:15.995743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:15.980683 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-home\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:16.019139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.010950 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tmp-dir\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:16.019139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.011207 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-model-cache\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:16.032858 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.032105 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:42:16.072826 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.047901 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:16.097086 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.097015 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-dshm\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:16.134741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.134152 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tls-certs\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:16.159797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.157939 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt9mw\" (UniqueName: \"kubernetes.io/projected/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-kube-api-access-wt9mw\") pod \"test-model-9e540d-kserve-66dc6b979c-58p4v\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:16.181743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.178206 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:16.181743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.178263 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:16.186740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.186115 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:16.186740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.186191 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:16.200747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.196867 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:16.789792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.789126 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:16.789792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:16.789204 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:17.056783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.050059 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:17.056783 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.050119 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:17.087018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.086975 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded" Nov 03 22:42:17.111709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.111668 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:17.126757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.121906 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:42:17.142884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.142474 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:42:17.182819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.143663 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:42:17.182819 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.148637 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/health\": context deadline exceeded" Nov 03 22:42:17.205511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.183996 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:17.205511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.184051 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:17.206693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.206342 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:17.219252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.219131 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:17.219252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.219187 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:17.467647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.432492 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 22:42:17.470276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.470244 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:17.470490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.470468 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:17.871883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.871850 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:17.880840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:17.880796 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:18.144793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.143818 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:18.196741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.178859 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 22:42:18.196741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.179489 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:42:18.259740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.251799 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:18.259740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.255733 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:18.259740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.255785 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:18.259740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.256201 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:18.259740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.256235 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:18.279741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.265965 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:42:18.376919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.334590 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v"] Nov 03 22:42:18.906738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.901063 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded" Nov 03 22:42:19.009745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.976673 4625 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:19.009745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.982875 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:42:19.009745 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:18.982949 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 22:42:19.129679 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.129413 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:19.129679 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.129486 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:19.161824 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.157435 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:42:19.212741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.192840 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:19.285757 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.275565 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:19.448803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.429744 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:19.509226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.507774 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:19.509226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.454730 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:42:19.509226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.507893 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:42:19.557103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.449868 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" podUID="ac4ab126-b50f-4673-8ff5-12444e47b637" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.119:50052\" within 1s: context deadline exceeded" Nov 03 22:42:19.591737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.467427 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:19.642740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.622707 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:19.643166 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.643124 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:19.749984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.749818 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:19.749984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.749876 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:19.804136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:19.804095 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" event={"ID":"1954d4e8-bf6d-4635-950b-dcd56a6f96f3","Type":"ContainerStarted","Data":"3090efe6d22b85bfbbe0028e39ccf73404fcb41efe694a5cc1ac12304f900082"} Nov 03 22:42:20.113756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.108046 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:20.113756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.108811 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:42:20.218751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.218273 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:20.241752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.220864 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.295937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.295876 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:20.296508 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.242429 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:20.296705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.242382 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.296883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.296863 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:20.416647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.416425 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:20.416647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.416536 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:42:20.417921 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.417494 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:42:20.419750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.419573 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:42:20.864212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.864175 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.864485 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.864453 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:21.040748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:20.990102 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 22:42:21.057165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.057130 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:42:21.113893 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.111363 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded" Nov 03 22:42:21.231816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.207321 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.372898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.221474 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.372898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.371395 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:21.372898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.371681 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:21.416814 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.416757 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:21.418460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.417691 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:21.801779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.801126 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.801779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.801193 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:22.015741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:21.900012 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn"] Nov 03 22:42:22.037749 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.028578 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 22:42:22.038012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.037991 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn"] Nov 03 22:42:22.047377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.032111 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.098740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.098396 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b1ec3a-openshift-default-dockercfg-p4w6w\"" Nov 03 22:42:22.150741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.149792 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded" Nov 03 22:42:22.187100 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.187065 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.194828 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.194786 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:22.222813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.189941 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.222813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.220846 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:22.338802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.305225 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-podinfo\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.340073 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.340017 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-envoy\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.358030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.349397 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-credential-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.358030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.349494 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.358030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.349569 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-token\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.358030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.349683 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-certs\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.358030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.349814 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2dqw\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-kube-api-access-g2dqw\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.358030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.349923 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-data\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.358030 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.349964 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istiod-ca-cert\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.418852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.418812 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.462858 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-certs\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.462980 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2dqw\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-kube-api-access-g2dqw\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.463062 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-data\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.463097 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istiod-ca-cert\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.463235 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-podinfo\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.463280 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-envoy\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.463347 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-credential-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.463397 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.463736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.463456 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-token\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.464406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.464258 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-certs\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.505754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.464815 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-data\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.505754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.465454 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istiod-ca-cert\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.505754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.467057 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-credential-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.505754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.489518 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.534127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.533682 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 03 22:42:22.601734 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.601631 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2dqw\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-kube-api-access-g2dqw\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.620772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.611375 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-envoy\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.658458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.658422 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-podinfo\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:22.765014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.764971 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.765217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.765039 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:22.769526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:22.769496 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-token\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:23.149742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.119814 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:23.190786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.189787 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.190786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.189853 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 22:42:23.215743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.213398 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded" Nov 03 22:42:23.215743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.214291 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:23.235741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.227123 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.235741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.227203 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:23.389742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.387654 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs"] Nov 03 22:42:23.397099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.397066 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.428780 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.427456 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:23.470743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.447423 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs"] Nov 03 22:42:23.519860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.484243 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fe7ca0-openshift-default-dockercfg-67bsj\"" Nov 03 22:42:23.691795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.628250 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-credential-socket\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.707739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.692162 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-podinfo\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.708190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.708161 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-certs\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.748862 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-data\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.746988 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.748943 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.749015 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-envoy\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.749115 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-socket\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.749187 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9758dc-7a16-4cab-bca6-3a3833037426-istiod-ca-cert\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.749256 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-token\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:23.749964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.749337 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqxgf\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-kube-api-access-qqxgf\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.031969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:23.991595 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 22:42:24.089444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.009248 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-credential-socket\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.090245 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.044062 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-credential-socket\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.090451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.090146 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-podinfo\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.090785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.090762 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-certs\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.090959 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.090935 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-data\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.091121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.091103 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-envoy\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.095838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.095817 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-socket\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.096060 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.096040 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-socket\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.096160 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.092286 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-data\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.096260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.096059 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9758dc-7a16-4cab-bca6-3a3833037426-istiod-ca-cert\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.096453 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.096435 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-token\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.096644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.096622 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqxgf\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-kube-api-access-qqxgf\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.096943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.096753 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9758dc-7a16-4cab-bca6-3a3833037426-istiod-ca-cert\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.097586 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.091676 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-certs\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.099120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.099071 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-podinfo\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.105529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.105260 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-envoy\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.156513 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.156480 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqxgf\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-kube-api-access-qqxgf\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.168662 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.168636 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:42:24.180538 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.180481 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-token\") pod \"e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.184792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.182164 4625 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.184792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.182216 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 22:42:24.184792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.182658 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 22:42:24.490265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.442874 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:24.490265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.465411 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:42:24.490265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.466422 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:42:24.502337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.493153 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:42:24.670040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.670005 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" event={"ID":"1954d4e8-bf6d-4635-950b-dcd56a6f96f3","Type":"ContainerStarted","Data":"e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516"} Nov 03 22:42:24.731923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.731871 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" podStartSLOduration=9.731856941 podStartE2EDuration="9.731856941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:24.722827734 +0000 UTC m=+1766.152163729" watchObservedRunningTime="2026-11-03 22:42:24.731856941 +0000 UTC m=+1766.161192963" Nov 03 22:42:24.783169 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.783135 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn"] Nov 03 22:42:24.788009 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:42:24.787978 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd03a2f_53f5_43cb_bb38_0d0aed39e22e.slice/crio-6d157f7087cdd1e276ca1b83343d57a3cd494233673c1ce83c54b5a60beb8bfb WatchSource:0}: Error finding container 6d157f7087cdd1e276ca1b83343d57a3cd494233673c1ce83c54b5a60beb8bfb: Status 404 returned error can't find the container with id 6d157f7087cdd1e276ca1b83343d57a3cd494233673c1ce83c54b5a60beb8bfb Nov 03 22:42:24.792235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.792194 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:24.792358 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.792284 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:24.792358 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.792328 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:24.835321 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.835287 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:42:24.838205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:24.838183 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:42:25.014275 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.014185 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs"] Nov 03 22:42:25.031925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.031888 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:25.032125 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.032110 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:25.032214 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.032203 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:42:25.180294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.180263 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:42:25.182208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.182174 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:42:25.205540 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.205513 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:25.681180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.681109 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" event={"ID":"9a9758dc-7a16-4cab-bca6-3a3833037426","Type":"ContainerStarted","Data":"cdcfa9957f998573419b9464f3595afd1139e32ca8a82159e9a59ff3c301f2e1"} Nov 03 22:42:25.681778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.681208 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" event={"ID":"9a9758dc-7a16-4cab-bca6-3a3833037426","Type":"ContainerStarted","Data":"7a66ac045b6d71b422b421f559d1892042a9770acfaf047a4afcaf371b9e7928"} Nov 03 22:42:25.683266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.683233 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" event={"ID":"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e","Type":"ContainerStarted","Data":"5b57d163b01b14a9f97954b1e9f582d5369436a0b8cb60b7d9991097fdde504d"} Nov 03 22:42:25.683413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.683285 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" event={"ID":"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e","Type":"ContainerStarted","Data":"6d157f7087cdd1e276ca1b83343d57a3cd494233673c1ce83c54b5a60beb8bfb"} Nov 03 22:42:25.728127 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.728062 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podStartSLOduration=2.728043434 podStartE2EDuration="2.728043434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:25.724369325 +0000 UTC m=+1767.153705318" watchObservedRunningTime="2026-11-03 22:42:25.728043434 +0000 UTC m=+1767.157379431" Nov 03 22:42:25.781775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:25.781723 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podStartSLOduration=4.781696944 podStartE2EDuration="4.781696944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:25.775330013 +0000 UTC m=+1767.204666006" watchObservedRunningTime="2026-11-03 22:42:25.781696944 +0000 UTC m=+1767.211032930" Nov 03 22:42:26.140096 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.140062 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:26.146743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.146531 4625 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:42:26.146743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.146604 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:42:26.205357 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.205325 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:26.219731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.219688 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:26.454002 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.453914 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:26.467260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.467225 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:26.692864 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.692822 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:26.693795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.693771 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:42:26.703817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:26.703792 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:42:27.142740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:27.138213 4625 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:42:27.142740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:27.138274 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:42:28.137511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:28.137077 4625 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.137511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:28.137141 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:42:29.139736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:29.138067 4625 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:42:29.139736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:29.138123 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:42:30.150450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:30.136126 4625 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:42:30.163881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:30.163830 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:42:31.144737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:31.137081 4625 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:42:31.144737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:31.137139 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 22:42:32.140114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:32.140081 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:32.140616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:32.140602 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:32.141421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:32.141400 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:42:32.190266 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:42:32.190229 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:42:45.749137 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.749097 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk"] Nov 03 22:42:45.756474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.756452 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.771541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.771510 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ccdb60-serving-cert\"" Nov 03 22:42:45.777906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.777881 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk"] Nov 03 22:42:45.864413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.864353 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-metrics-certs\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.864680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.864426 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-maas-api-tls\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.864680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.864581 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7589j\" (UniqueName: \"kubernetes.io/projected/8cca9d4d-9923-47ac-8c95-96ac0376b42f-kube-api-access-7589j\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.864680 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.864664 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cca9d4d-9923-47ac-8c95-96ac0376b42f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.976686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.966783 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cca9d4d-9923-47ac-8c95-96ac0376b42f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.976686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.967194 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-metrics-certs\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.976686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.967253 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-maas-api-tls\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.976686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.967406 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7589j\" (UniqueName: \"kubernetes.io/projected/8cca9d4d-9923-47ac-8c95-96ac0376b42f-kube-api-access-7589j\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:45.976686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:45.967597 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cca9d4d-9923-47ac-8c95-96ac0376b42f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:46.003907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:46.001507 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-metrics-certs\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:46.003907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:46.002258 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-maas-api-tls\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:46.008743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:46.007090 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7589j\" (UniqueName: \"kubernetes.io/projected/8cca9d4d-9923-47ac-8c95-96ac0376b42f-kube-api-access-7589j\") pod \"maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:46.089041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:46.088998 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:46.473112 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:42:46.473071 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cca9d4d_9923_47ac_8c95_96ac0376b42f.slice/crio-39cb02f9f334421339cc2911f46a1d8c5e313f5e20c20b841c006058567d9b67 WatchSource:0}: Error finding container 39cb02f9f334421339cc2911f46a1d8c5e313f5e20c20b841c006058567d9b67: Status 404 returned error can't find the container with id 39cb02f9f334421339cc2911f46a1d8c5e313f5e20c20b841c006058567d9b67 Nov 03 22:42:46.473898 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:46.473856 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk"] Nov 03 22:42:47.099024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:47.098994 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" event={"ID":"8cca9d4d-9923-47ac-8c95-96ac0376b42f","Type":"ContainerStarted","Data":"39cb02f9f334421339cc2911f46a1d8c5e313f5e20c20b841c006058567d9b67"} Nov 03 22:42:48.110642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:48.110608 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" event={"ID":"8cca9d4d-9923-47ac-8c95-96ac0376b42f","Type":"ContainerStarted","Data":"beebd4d19186ae0563f7c293b5c68a502f206ffbf1376785e3059cb49042858b"} Nov 03 22:42:48.111161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:48.110708 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:42:48.155284 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:48.155224 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" podStartSLOduration=2.593676836 podStartE2EDuration="3.155207717s" totalImagesPullingTime="561.530881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:48.150013583 +0000 UTC m=+1789.579349590" watchObservedRunningTime="2026-11-03 22:42:48.155207717 +0000 UTC m=+1789.584543740" Nov 03 22:42:54.125430 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:42:54.125400 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:43:02.186481 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:43:02.186437 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:43:02.375742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:02.375698 4625 generic.go:356] "Generic (PLEG): container finished" podID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerID="e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558" exitCode=0 Nov 03 22:43:02.375932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:02.375749 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" event={"ID":"22fede61-3e56-4ddf-baa0-cfa2359c4556","Type":"ContainerDied","Data":"e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558"} Nov 03 22:43:02.376190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:02.376173 4625 scope.go:122] "RemoveContainer" containerID="e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558" Nov 03 22:43:02.378183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:02.378148 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:43:02.378294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:02.378220 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:43:02.378294 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:02.378252 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:43:03.465739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:03.462895 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" event={"ID":"22fede61-3e56-4ddf-baa0-cfa2359c4556","Type":"ContainerStarted","Data":"24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e"} Nov 03 22:43:03.518179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:03.518149 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:43:03.548004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:03.541757 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:43:03.973189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:03.973153 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:43:03.974081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:03.974051 4625 patch_prober.go:29] interesting pod/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:43:03.974219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:03.974112 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:43:04.972907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:04.972869 4625 patch_prober.go:29] interesting pod/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:43:04.973343 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:04.972936 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:43:05.987926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:05.987842 4625 patch_prober.go:29] interesting pod/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:43:05.987926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:05.987908 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:43:06.973843 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:06.973800 4625 patch_prober.go:29] interesting pod/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:43:06.974056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:06.973876 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:43:07.973635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:07.972768 4625 patch_prober.go:29] interesting pod/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.973635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:07.972818 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:43:08.972834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:08.972787 4625 patch_prober.go:29] interesting pod/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.973051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:08.972852 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 22:43:09.979613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:09.979584 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:43:09.980111 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:09.980098 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:43:09.980962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:09.980940 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:43:13.664966 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.664930 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22"] Nov 03 22:43:13.670806 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.670777 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.693834 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.693064 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fe7ca0-serving-cert\"" Nov 03 22:43:13.710036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.709686 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22"] Nov 03 22:43:13.776332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.776300 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.776542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.776510 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p94k\" (UniqueName: \"kubernetes.io/projected/eaf35d5f-1526-42ce-b8af-1f400197dc3d-kube-api-access-2p94k\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.776612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.776559 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-metrics-certs\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.776771 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.776755 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf35d5f-1526-42ce-b8af-1f400197dc3d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.881874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.881828 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf35d5f-1526-42ce-b8af-1f400197dc3d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.907461 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.907429 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.909624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.908111 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p94k\" (UniqueName: \"kubernetes.io/projected/eaf35d5f-1526-42ce-b8af-1f400197dc3d-kube-api-access-2p94k\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.930730 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:43:13.907792 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-serving-cert: secret "maas-api-e2e-inf-b-fe7ca0-serving-cert" not found Nov 03 22:43:13.931595 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:43:13.931576 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls podName:eaf35d5f-1526-42ce-b8af-1f400197dc3d nodeName:}" failed. No retries permitted until 2026-11-03 22:43:14.431545827 +0000 UTC m=+1815.860881811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls") pod "maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" (UID: "eaf35d5f-1526-42ce-b8af-1f400197dc3d") : secret "maas-api-e2e-inf-b-fe7ca0-serving-cert" not found Nov 03 22:43:13.931994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.882839 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf35d5f-1526-42ce-b8af-1f400197dc3d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.932174 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.930619 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-metrics-certs\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.972028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.962408 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p94k\" (UniqueName: \"kubernetes.io/projected/eaf35d5f-1526-42ce-b8af-1f400197dc3d-kube-api-access-2p94k\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:13.977735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:13.973841 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-metrics-certs\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:14.443407 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:14.443372 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:14.445879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:14.445853 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:14.588078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:14.588046 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:14.988590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:14.988553 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22"] Nov 03 22:43:16.140742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:16.130426 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" event={"ID":"eaf35d5f-1526-42ce-b8af-1f400197dc3d","Type":"ContainerStarted","Data":"efe053106a8d99151a2aad1f1434486c3020eef297a128d91ccce52f1f5b4e72"} Nov 03 22:43:16.140742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:16.130506 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" event={"ID":"eaf35d5f-1526-42ce-b8af-1f400197dc3d","Type":"ContainerStarted","Data":"38faf7f7d3dec3bc2694d5c041f5341b2df1a80bc6498a7ea61ce0afd5634705"} Nov 03 22:43:16.140742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:16.132135 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:16.186502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:16.186434 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" podStartSLOduration=2.900572244 podStartE2EDuration="3.186413311s" totalImagesPullingTime="285.841067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:16.178832163 +0000 UTC m=+1817.608168157" watchObservedRunningTime="2026-11-03 22:43:16.186413311 +0000 UTC m=+1817.615749303" Nov 03 22:43:16.996216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:16.996174 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h"] Nov 03 22:43:16.996542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:16.996435 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" containerID="cri-o://9b1718bfeb88f6a969ac2cbb857b9c531b82d0d43e67e83eaffce908a66e8d88" gracePeriod=30 Nov 03 22:43:21.827136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:21.827098 4625 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:43:21.827544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:21.827155 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:43:22.290192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.290150 4625 generic.go:356] "Generic (PLEG): container finished" podID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerID="9b1718bfeb88f6a969ac2cbb857b9c531b82d0d43e67e83eaffce908a66e8d88" exitCode=0 Nov 03 22:43:22.290375 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.290250 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" event={"ID":"d60a3457-94ee-4bb6-a9f8-7d32d0b63378","Type":"ContainerDied","Data":"9b1718bfeb88f6a969ac2cbb857b9c531b82d0d43e67e83eaffce908a66e8d88"} Nov 03 22:43:22.290505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.290380 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" event={"ID":"d60a3457-94ee-4bb6-a9f8-7d32d0b63378","Type":"ContainerDied","Data":"4e242f65a1cff7f2facf13f6380586cf63fc5bac97d2a73fa03cfb9cbfa9e4b1"} Nov 03 22:43:22.290505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.290418 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e242f65a1cff7f2facf13f6380586cf63fc5bac97d2a73fa03cfb9cbfa9e4b1" Nov 03 22:43:22.312611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.312584 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:43:22.392072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.391236 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-envoy\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.392072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.391374 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-kube-api-access-trs76\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-kube-api-access-trs76\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.392072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.391453 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istiod-ca-cert\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.392072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.391559 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-socket\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.392072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.391625 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-certs\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.392072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.391652 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-credential-socket\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.392072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.391698 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-token\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-token\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.392449 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.392214 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istiod-ca-cert" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:22.393468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.393442 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-credential-socket" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.393614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.393579 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-certs" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.394229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.394198 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-socket" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.394352 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.394230 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-podinfo\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.394426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.394356 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-data\") pod \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\" (UID: \"d60a3457-94ee-4bb6-a9f8-7d32d0b63378\") " Nov 03 22:43:22.394637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.394615 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-data" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.395464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.395437 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.395464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.395465 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.395656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.395479 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.395656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.395492 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.395656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.395506 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.401759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.396597 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-token" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:22.401759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.396628 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-kube-api-access-trs76" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "kube-api-access-trs76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:22.401759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.396681 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-envoy" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.401759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.396937 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-podinfo" pod "d60a3457-94ee-4bb6-a9f8-7d32d0b63378" (UID: "d60a3457-94ee-4bb6-a9f8-7d32d0b63378"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:22.497754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.497672 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.497754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.497704 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.497754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.497731 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trs76\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-kube-api-access-trs76\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.497754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:22.497747 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60a3457-94ee-4bb6-a9f8-7d32d0b63378-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:43:23.241041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:23.241013 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:43:23.302073 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:23.302041 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h" Nov 03 22:43:23.361028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:23.360990 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h"] Nov 03 22:43:23.372958 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:23.372924 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-cqz7h"] Nov 03 22:43:24.738919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:24.738888 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" path="/var/lib/kubelet/pods/d60a3457-94ee-4bb6-a9f8-7d32d0b63378/volumes" Nov 03 22:43:32.188581 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:43:32.188538 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:43:45.971541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:45.971446 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2"] Nov 03 22:43:45.972872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:45.972845 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" Nov 03 22:43:45.972872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:45.972869 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" Nov 03 22:43:45.973122 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:45.973069 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60a3457-94ee-4bb6-a9f8-7d32d0b63378" containerName="istio-proxy" Nov 03 22:43:45.988736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:45.978966 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:45.999697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:45.999667 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2"] Nov 03 22:43:45.999943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:45.999755 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-963936-serving-cert\"" Nov 03 22:43:46.075926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.075887 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5m79\" (UniqueName: \"kubernetes.io/projected/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-kube-api-access-k5m79\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.076187 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.076167 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.076349 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.076335 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-metrics-certs\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.076571 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.076557 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.188106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.179231 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5m79\" (UniqueName: \"kubernetes.io/projected/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-kube-api-access-k5m79\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.188106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.185405 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.188106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.185524 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-metrics-certs\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.188106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.185703 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.188106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.186704 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.188106 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:43:46.186835 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-serving-cert: secret "maas-api-e2e-inf-b-963936-serving-cert" not found Nov 03 22:43:46.188106 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:43:46.186900 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls podName:34857ca0-8bb7-4b57-a795-35aaa7c4ba65 nodeName:}" failed. No retries permitted until 2026-11-03 22:43:46.686881804 +0000 UTC m=+1848.116217780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls") pod "maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" (UID: "34857ca0-8bb7-4b57-a795-35aaa7c4ba65") : secret "maas-api-e2e-inf-b-963936-serving-cert" not found Nov 03 22:43:46.191444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.190161 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-metrics-certs\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.266777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.266478 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5m79\" (UniqueName: \"kubernetes.io/projected/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-kube-api-access-k5m79\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.703530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.703485 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.706070 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.706043 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls\") pod \"maas-api-e2e-inf-b-963936-5f5c97895c-sctt2\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:46.901789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:46.901749 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:47.296872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:47.296636 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2"] Nov 03 22:43:47.297554 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:43:47.297527 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34857ca0_8bb7_4b57_a795_35aaa7c4ba65.slice/crio-5d736094788d944953e8d10b5f5cf0ab03e978b0c068cb6c75dc900529991ab9 WatchSource:0}: Error finding container 5d736094788d944953e8d10b5f5cf0ab03e978b0c068cb6c75dc900529991ab9: Status 404 returned error can't find the container with id 5d736094788d944953e8d10b5f5cf0ab03e978b0c068cb6c75dc900529991ab9 Nov 03 22:43:47.581599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:47.581507 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" event={"ID":"34857ca0-8bb7-4b57-a795-35aaa7c4ba65","Type":"ContainerStarted","Data":"5d736094788d944953e8d10b5f5cf0ab03e978b0c068cb6c75dc900529991ab9"} Nov 03 22:43:48.591364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:48.591323 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" event={"ID":"34857ca0-8bb7-4b57-a795-35aaa7c4ba65","Type":"ContainerStarted","Data":"dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a"} Nov 03 22:43:48.591788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:48.591416 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:48.633683 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:48.633631 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" podStartSLOduration=3.306117897 podStartE2EDuration="3.633617255s" totalImagesPullingTime="327.499358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:48.632030081 +0000 UTC m=+1850.061366099" watchObservedRunningTime="2026-11-03 22:43:48.633617255 +0000 UTC m=+1850.062953245" Nov 03 22:43:54.604580 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:54.604552 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:43:55.754219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.754178 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns"] Nov 03 22:43:55.760419 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.760390 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.774958 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.774927 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns"] Nov 03 22:43:55.775258 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.775233 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-124490-serving-cert\"" Nov 03 22:43:55.875320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.875285 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94kfv\" (UniqueName: \"kubernetes.io/projected/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-kube-api-access-94kfv\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.875521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.875449 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-openshift-service-ca\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.875606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.875581 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-metrics-certs\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.875668 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.875623 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-maas-api-tls\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.987810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.980012 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94kfv\" (UniqueName: \"kubernetes.io/projected/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-kube-api-access-94kfv\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.987810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.980206 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-openshift-service-ca\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.987810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.980280 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-metrics-certs\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.987810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.980351 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-maas-api-tls\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:55.987810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:55.982310 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-openshift-service-ca\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:56.019426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:56.019345 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-maas-api-tls\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:56.025437 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:56.025367 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94kfv\" (UniqueName: \"kubernetes.io/projected/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-kube-api-access-94kfv\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:56.030772 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:56.029133 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-metrics-certs\") pod \"maas-api-e2e-inf-b-124490-67cf698669-kx9ns\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:56.091293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:56.091258 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:56.410119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:56.410088 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns"] Nov 03 22:43:56.410677 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:43:56.410657 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb37d5be_770b_46cf_a9d7_ec7d4c6c4b35.slice/crio-150b215b11025d2ce58940989e19f20ac8f95fa613b8c6f4895716ccbccdcbb8 WatchSource:0}: Error finding container 150b215b11025d2ce58940989e19f20ac8f95fa613b8c6f4895716ccbccdcbb8: Status 404 returned error can't find the container with id 150b215b11025d2ce58940989e19f20ac8f95fa613b8c6f4895716ccbccdcbb8 Nov 03 22:43:56.711567 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:56.711495 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" event={"ID":"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35","Type":"ContainerStarted","Data":"150b215b11025d2ce58940989e19f20ac8f95fa613b8c6f4895716ccbccdcbb8"} Nov 03 22:43:57.721958 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:57.721921 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" event={"ID":"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35","Type":"ContainerStarted","Data":"0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339"} Nov 03 22:43:57.722364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:57.721979 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:43:57.774810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:43:57.774758 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" podStartSLOduration=2.453864263 podStartE2EDuration="2.774742182s" totalImagesPullingTime="320.877919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:57.769050157 +0000 UTC m=+1859.198386150" watchObservedRunningTime="2026-11-03 22:43:57.774742182 +0000 UTC m=+1859.204078174" Nov 03 22:44:02.189544 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:44:02.189499 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:44:03.736198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:03.736168 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:44:06.338452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:06.338421 4625 scope.go:122] "RemoveContainer" containerID="9b1718bfeb88f6a969ac2cbb857b9c531b82d0d43e67e83eaffce908a66e8d88" Nov 03 22:44:12.733872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:12.733830 4625 patch_prober.go:29] interesting pod/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:12.734393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:12.733896 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:15.075760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:15.075704 4625 generic.go:356] "Generic (PLEG): container finished" podID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerID="cdcfa9957f998573419b9464f3595afd1139e32ca8a82159e9a59ff3c301f2e1" exitCode=0 Nov 03 22:44:15.076325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:15.075763 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" event={"ID":"9a9758dc-7a16-4cab-bca6-3a3833037426","Type":"ContainerDied","Data":"cdcfa9957f998573419b9464f3595afd1139e32ca8a82159e9a59ff3c301f2e1"} Nov 03 22:44:15.076325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:15.076298 4625 scope.go:122] "RemoveContainer" containerID="cdcfa9957f998573419b9464f3595afd1139e32ca8a82159e9a59ff3c301f2e1" Nov 03 22:44:15.078515 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:15.078484 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:44:15.078636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:15.078556 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:44:15.078636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:15.078589 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:44:16.089916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:16.089877 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" event={"ID":"9a9758dc-7a16-4cab-bca6-3a3833037426","Type":"ContainerStarted","Data":"e5d3792e36390b60d3409c67ca4a394ad2f60f55a7136cf7bacdba903524f0ac"} Nov 03 22:44:16.454802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:16.454174 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:44:16.454802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:16.454297 4625 patch_prober.go:29] interesting pod/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:44:16.454802 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:16.454340 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:44:17.450751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:17.448147 4625 patch_prober.go:29] interesting pod/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:44:17.450751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:17.448206 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:44:18.445079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:18.445041 4625 patch_prober.go:29] interesting pod/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:44:18.445382 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:18.445351 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:44:19.444105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:19.444065 4625 patch_prober.go:29] interesting pod/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:44:19.444709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:19.444675 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:44:20.447146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:20.447108 4625 patch_prober.go:29] interesting pod/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:44:20.447840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:20.447806 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 22:44:21.503809 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:21.503157 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:44:21.504302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:21.504161 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:44:21.505139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:44:21.505112 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:44:32.187420 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:44:32.187376 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:45:02.187000 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:45:02.186964 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:45:32.188341 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:45:32.188261 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:46:02.190784 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:46:02.190740 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:46:21.047056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.047015 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk"] Nov 03 22:46:21.047660 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.047288 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" podUID="613cb909-ee79-4d35-ab13-e290d8736651" containerName="maas-api" containerID="cri-o://f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047" gracePeriod=30 Nov 03 22:46:21.688738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.683975 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:46:21.877522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.864292 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613cb909-ee79-4d35-ab13-e290d8736651-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613cb909-ee79-4d35-ab13-e290d8736651-openshift-service-ca\") pod \"613cb909-ee79-4d35-ab13-e290d8736651\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " Nov 03 22:46:21.877522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.864430 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-metrics-certs\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-metrics-certs\") pod \"613cb909-ee79-4d35-ab13-e290d8736651\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " Nov 03 22:46:21.877522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.864612 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls\") pod \"613cb909-ee79-4d35-ab13-e290d8736651\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " Nov 03 22:46:21.877522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.864654 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613cb909-ee79-4d35-ab13-e290d8736651-kube-api-access-7tqdv\" (UniqueName: \"kubernetes.io/projected/613cb909-ee79-4d35-ab13-e290d8736651-kube-api-access-7tqdv\") pod \"613cb909-ee79-4d35-ab13-e290d8736651\" (UID: \"613cb909-ee79-4d35-ab13-e290d8736651\") " Nov 03 22:46:21.877522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.866096 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613cb909-ee79-4d35-ab13-e290d8736651-openshift-service-ca" pod "613cb909-ee79-4d35-ab13-e290d8736651" (UID: "613cb909-ee79-4d35-ab13-e290d8736651"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:21.911853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.908884 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613cb909-ee79-4d35-ab13-e290d8736651-kube-api-access-7tqdv" pod "613cb909-ee79-4d35-ab13-e290d8736651" (UID: "613cb909-ee79-4d35-ab13-e290d8736651"). InnerVolumeSpecName "kube-api-access-7tqdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:21.911853 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.908994 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls" pod "613cb909-ee79-4d35-ab13-e290d8736651" (UID: "613cb909-ee79-4d35-ab13-e290d8736651"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:21.964195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.964133 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-metrics-certs" pod "613cb909-ee79-4d35-ab13-e290d8736651" (UID: "613cb909-ee79-4d35-ab13-e290d8736651"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:21.967849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.967817 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613cb909-ee79-4d35-ab13-e290d8736651-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:21.967849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.967852 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:21.968161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.967867 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613cb909-ee79-4d35-ab13-e290d8736651-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:21.968161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:21.967880 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tqdv\" (UniqueName: \"kubernetes.io/projected/613cb909-ee79-4d35-ab13-e290d8736651-kube-api-access-7tqdv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:22.154091 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.153994 4625 generic.go:356] "Generic (PLEG): container finished" podID="613cb909-ee79-4d35-ab13-e290d8736651" containerID="f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047" exitCode=0 Nov 03 22:46:22.154555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.154089 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" Nov 03 22:46:22.154555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.154324 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" event={"ID":"613cb909-ee79-4d35-ab13-e290d8736651","Type":"ContainerDied","Data":"f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047"} Nov 03 22:46:22.154555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.154416 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk" event={"ID":"613cb909-ee79-4d35-ab13-e290d8736651","Type":"ContainerDied","Data":"0bde2b3e861e04fff5c691770223b0c3ec29b54406ba6773cc66c8e1d2312003"} Nov 03 22:46:22.154555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.154445 4625 scope.go:122] "RemoveContainer" containerID="f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047" Nov 03 22:46:22.169684 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.169658 4625 scope.go:122] "RemoveContainer" containerID="f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047" Nov 03 22:46:22.169999 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:46:22.169974 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047\": container with ID starting with f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047 not found: ID does not exist" containerID="f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047" Nov 03 22:46:22.170073 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.170011 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047"} err="failed to get container status \"f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047\": rpc error: code = NotFound desc = could not find container \"f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047\": container with ID starting with f01a95caaa5495bf65b74cb9a2b6fa309f203908066c0002bf159b8cdd390047 not found: ID does not exist" Nov 03 22:46:22.209226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.209198 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk"] Nov 03 22:46:22.220186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.220153 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4b5e2b-5ff54bb48-cgssk"] Nov 03 22:46:22.730642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:22.730610 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613cb909-ee79-4d35-ab13-e290d8736651" path="/var/lib/kubelet/pods/613cb909-ee79-4d35-ab13-e290d8736651/volumes" Nov 03 22:46:32.191353 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:46:32.191308 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:46:47.829652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:47.829556 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l"] Nov 03 22:46:47.830178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:47.829894 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" containerID="cri-o://95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1" gracePeriod=30 Nov 03 22:46:48.228874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:48.228839 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t"] Nov 03 22:46:48.229160 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:48.229082 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" containerID="cri-o://28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1" gracePeriod=30 Nov 03 22:46:52.408766 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:52.408708 4625 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:46:52.409183 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:52.408822 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:46:52.927505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:52.927476 4625 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:46:52.927675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:52.927532 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:46:53.127429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.127403 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:46:53.178509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.178469 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-certs\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.178698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.178607 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-socket\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.178811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.178764 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-kube-api-access-wlcs4\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-kube-api-access-wlcs4\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.178879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.178817 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-credential-socket\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.178879 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.178818 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-certs" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.179001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.178909 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-socket" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.179001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.178937 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istiod-ca-cert\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.179109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.179015 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-envoy\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.179109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.179039 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-token\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-token\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.179109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.179086 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-credential-socket" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.179274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.179116 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-podinfo\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.179274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.179240 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-data\") pod \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\" (UID: \"d5b65df3-8781-4fc4-be9d-8c1df26fa290\") " Nov 03 22:46:53.179385 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.179268 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istiod-ca-cert" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:53.179610 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.179585 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-data" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.180105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.180085 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.180205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.180112 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.180205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.180128 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.180205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.180141 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.180205 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.180154 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.181364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.181338 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-envoy" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.181611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.181585 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-kube-api-access-wlcs4" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "kube-api-access-wlcs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:53.181731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.181623 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-token" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:53.181731 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.181668 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-podinfo" pod "d5b65df3-8781-4fc4-be9d-8c1df26fa290" (UID: "d5b65df3-8781-4fc4-be9d-8c1df26fa290"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:46:53.281251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.281214 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlcs4\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-kube-api-access-wlcs4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.281251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.281253 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.281425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.281262 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.281425 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.281270 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b65df3-8781-4fc4-be9d-8c1df26fa290-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.500548 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.500523 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:46:53.536823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.536793 4625 generic.go:356] "Generic (PLEG): container finished" podID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerID="28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1" exitCode=0 Nov 03 22:46:53.536984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.536859 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" Nov 03 22:46:53.536984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.536872 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" event={"ID":"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43","Type":"ContainerDied","Data":"28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1"} Nov 03 22:46:53.536984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.536932 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t" event={"ID":"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43","Type":"ContainerDied","Data":"fac83083e0858d3d79c2f0e3a144047d206444bff1195b93e708c0f84fc3dd4d"} Nov 03 22:46:53.536984 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.536964 4625 scope.go:122] "RemoveContainer" containerID="28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1" Nov 03 22:46:53.538651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.538569 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" Nov 03 22:46:53.538651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.538594 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" event={"ID":"d5b65df3-8781-4fc4-be9d-8c1df26fa290","Type":"ContainerDied","Data":"95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1"} Nov 03 22:46:53.538844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.538513 4625 generic.go:356] "Generic (PLEG): container finished" podID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerID="95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1" exitCode=0 Nov 03 22:46:53.538844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.538798 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l" event={"ID":"d5b65df3-8781-4fc4-be9d-8c1df26fa290","Type":"ContainerDied","Data":"143199237e6f9407532339b3cd18e4eecdfdd14cfc153a644b544ab37c6d4455"} Nov 03 22:46:53.552693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.552676 4625 scope.go:122] "RemoveContainer" containerID="9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0" Nov 03 22:46:53.569938 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.569916 4625 scope.go:122] "RemoveContainer" containerID="28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1" Nov 03 22:46:53.570251 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:46:53.570215 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1\": container with ID starting with 28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1 not found: ID does not exist" containerID="28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1" Nov 03 22:46:53.570356 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.570254 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1"} err="failed to get container status \"28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1\": rpc error: code = NotFound desc = could not find container \"28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1\": container with ID starting with 28fa28441df2d0c774cbaf361e360947ae7a44e8005abe49d44ed83b026043c1 not found: ID does not exist" Nov 03 22:46:53.570356 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.570277 4625 scope.go:122] "RemoveContainer" containerID="9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0" Nov 03 22:46:53.570543 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:46:53.570521 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0\": container with ID starting with 9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0 not found: ID does not exist" containerID="9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0" Nov 03 22:46:53.570618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.570549 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0"} err="failed to get container status \"9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0\": rpc error: code = NotFound desc = could not find container \"9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0\": container with ID starting with 9958312c0d074c46e9850b42abee18982596e375f43eb98f02a5e5b781bde8e0 not found: ID does not exist" Nov 03 22:46:53.570618 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.570567 4625 scope.go:122] "RemoveContainer" containerID="95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1" Nov 03 22:46:53.585087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.585070 4625 scope.go:122] "RemoveContainer" containerID="95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1" Nov 03 22:46:53.585398 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:46:53.585369 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1\": container with ID starting with 95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1 not found: ID does not exist" containerID="95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1" Nov 03 22:46:53.585483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.585406 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1"} err="failed to get container status \"95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1\": rpc error: code = NotFound desc = could not find container \"95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1\": container with ID starting with 95192c933548b7af4530938216dd7139fd20b48ec1ff4f7bdd27c3d97db90ae1 not found: ID does not exist" Nov 03 22:46:53.585483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.585377 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-certs\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.585565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.585518 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-kube-api-access-q29q4\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-kube-api-access-q29q4\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.585644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.585618 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-certs" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.586051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586026 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-podinfo\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.586165 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586151 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-token\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-token\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.586230 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586204 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-envoy\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.586967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586314 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istiod-ca-cert\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.586967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586358 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-socket\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.586967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586400 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-credential-socket\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.586967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586456 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-data\") pod \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\" (UID: \"adb7bc4c-db84-4ea3-b6dd-75e2d983cd43\") " Nov 03 22:46:53.586967 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.586789 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-credential-socket" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.587317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.587289 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.587432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.587323 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.587432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.587321 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-data" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.588651 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.588625 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istiod-ca-cert" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:53.588773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.588700 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-socket" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.589063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.589040 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-podinfo" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:46:53.589172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.589068 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-kube-api-access-q29q4" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "kube-api-access-q29q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:53.589319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.589295 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-token" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:53.590484 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.590461 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-envoy" pod "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" (UID: "adb7bc4c-db84-4ea3-b6dd-75e2d983cd43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:53.625747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.625669 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l"] Nov 03 22:46:53.631862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.631834 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-gwb2l"] Nov 03 22:46:53.689223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.689141 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.689223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.689171 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.689223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.689180 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29q4\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-kube-api-access-q29q4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.689223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.689189 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.689223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.689199 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.689223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.689206 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.689223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.689213 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:46:53.906007 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.905975 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t"] Nov 03 22:46:53.910777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:53.910750 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-dcb6t"] Nov 03 22:46:54.729915 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:54.729884 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" path="/var/lib/kubelet/pods/adb7bc4c-db84-4ea3-b6dd-75e2d983cd43/volumes" Nov 03 22:46:54.730402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:46:54.730388 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" path="/var/lib/kubelet/pods/d5b65df3-8781-4fc4-be9d-8c1df26fa290/volumes" Nov 03 22:47:02.188025 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:47:02.187980 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:47:04.551339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:04.551300 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2"] Nov 03 22:47:19.181693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.181656 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s"] Nov 03 22:47:19.182474 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.182443 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" containerID="cri-o://4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3" gracePeriod=30 Nov 03 22:47:19.627923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.627223 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" containerID="cri-o://a25371e603f7739605b2f9c14423258a3c47a9ef57ce5840bb348573f1baf9b0" gracePeriod=45 Nov 03 22:47:19.873267 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.873237 4625 generic.go:356] "Generic (PLEG): container finished" podID="c183151e-5e97-4008-a434-7997f7cec128" containerID="a25371e603f7739605b2f9c14423258a3c47a9ef57ce5840bb348573f1baf9b0" exitCode=0 Nov 03 22:47:19.873449 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.873316 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" event={"ID":"c183151e-5e97-4008-a434-7997f7cec128","Type":"ContainerDied","Data":"a25371e603f7739605b2f9c14423258a3c47a9ef57ce5840bb348573f1baf9b0"} Nov 03 22:47:19.960779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.960735 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:47:19.992678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.992647 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c183151e-5e97-4008-a434-7997f7cec128-kube-api-access-nnjgq\" (UniqueName: \"kubernetes.io/projected/c183151e-5e97-4008-a434-7997f7cec128-kube-api-access-nnjgq\") pod \"c183151e-5e97-4008-a434-7997f7cec128\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " Nov 03 22:47:19.992863 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.992810 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-tmp-dir\") pod \"c183151e-5e97-4008-a434-7997f7cec128\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " Nov 03 22:47:19.992925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.992897 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c183151e-5e97-4008-a434-7997f7cec128-tls-certs\" (UniqueName: \"kubernetes.io/secret/c183151e-5e97-4008-a434-7997f7cec128-tls-certs\") pod \"c183151e-5e97-4008-a434-7997f7cec128\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " Nov 03 22:47:19.992974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.992940 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-model-cache\") pod \"c183151e-5e97-4008-a434-7997f7cec128\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " Nov 03 22:47:19.993076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993050 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-tmp-dir" pod "c183151e-5e97-4008-a434-7997f7cec128" (UID: "c183151e-5e97-4008-a434-7997f7cec128"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:19.993141 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993126 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-home\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-home\") pod \"c183151e-5e97-4008-a434-7997f7cec128\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " Nov 03 22:47:19.993202 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993146 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-dshm\") pod \"c183151e-5e97-4008-a434-7997f7cec128\" (UID: \"c183151e-5e97-4008-a434-7997f7cec128\") " Nov 03 22:47:19.993278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993252 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-model-cache" pod "c183151e-5e97-4008-a434-7997f7cec128" (UID: "c183151e-5e97-4008-a434-7997f7cec128"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:19.993351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993335 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-home" pod "c183151e-5e97-4008-a434-7997f7cec128" (UID: "c183151e-5e97-4008-a434-7997f7cec128"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:19.993818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993797 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:19.993818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993815 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:19.993818 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.993823 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:19.995061 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.995040 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c183151e-5e97-4008-a434-7997f7cec128-kube-api-access-nnjgq" pod "c183151e-5e97-4008-a434-7997f7cec128" (UID: "c183151e-5e97-4008-a434-7997f7cec128"). InnerVolumeSpecName "kube-api-access-nnjgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:19.995154 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.995085 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-dshm" pod "c183151e-5e97-4008-a434-7997f7cec128" (UID: "c183151e-5e97-4008-a434-7997f7cec128"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:19.995154 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:19.995090 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c183151e-5e97-4008-a434-7997f7cec128-tls-certs" pod "c183151e-5e97-4008-a434-7997f7cec128" (UID: "c183151e-5e97-4008-a434-7997f7cec128"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:20.095032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.094997 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjgq\" (UniqueName: \"kubernetes.io/projected/c183151e-5e97-4008-a434-7997f7cec128-kube-api-access-nnjgq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:20.095032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.095034 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183151e-5e97-4008-a434-7997f7cec128-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:20.095032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.095048 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183151e-5e97-4008-a434-7997f7cec128-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:20.884229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.884199 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" Nov 03 22:47:20.884229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.884221 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2" event={"ID":"c183151e-5e97-4008-a434-7997f7cec128","Type":"ContainerDied","Data":"92e136419e400381c4f76114b58886de862d17ba984658637a9f1d99439fb0c1"} Nov 03 22:47:20.884815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.884266 4625 scope.go:122] "RemoveContainer" containerID="a25371e603f7739605b2f9c14423258a3c47a9ef57ce5840bb348573f1baf9b0" Nov 03 22:47:20.938949 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.938913 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2"] Nov 03 22:47:20.948762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:20.948732 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b1ec3a/test-model-b1ec3a-kserve-566b9f4fbf-85tx2"] Nov 03 22:47:22.730531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:22.730494 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c183151e-5e97-4008-a434-7997f7cec128" path="/var/lib/kubelet/pods/c183151e-5e97-4008-a434-7997f7cec128/volumes" Nov 03 22:47:23.982616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:23.982583 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ddx5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:23.983066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:23.982644 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:24.488198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.488133 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:47:24.594648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.594618 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-envoy\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.595555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.595024 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61e5926-471e-4f07-ae43-c988b094340c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e5926-471e-4f07-ae43-c988b094340c-istio-podinfo\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.597424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.597399 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61e5926-471e-4f07-ae43-c988b094340c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e5926-471e-4f07-ae43-c988b094340c-istiod-ca-cert\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.597656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.597452 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-envoy" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:24.598102 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.597822 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61e5926-471e-4f07-ae43-c988b094340c-istiod-ca-cert" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:24.598247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.597895 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61e5926-471e-4f07-ae43-c988b094340c-istio-podinfo" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:24.598350 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.598039 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-socket" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:24.598475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.598458 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-socket\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.598661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.598648 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-certs\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.599178 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.599159 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-istio-token\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-istio-token\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.600167 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.599089 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-certs" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:24.600424 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.600400 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-kube-api-access-bz6pl\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-kube-api-access-bz6pl\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.600896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.600864 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-credential-socket\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.601647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.600917 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-data\") pod \"b61e5926-471e-4f07-ae43-c988b094340c\" (UID: \"b61e5926-471e-4f07-ae43-c988b094340c\") " Nov 03 22:47:24.601647 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.601602 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-credential-socket" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:24.601861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.601776 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-data" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:24.602785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.602764 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e5926-471e-4f07-ae43-c988b094340c-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.602785 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.602785 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.602953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.602809 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.602953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.602822 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.602953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.602835 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.602953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.602848 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e5926-471e-4f07-ae43-c988b094340c-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.602953 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.602860 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e5926-471e-4f07-ae43-c988b094340c-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.611527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.606243 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-istio-token" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:24.611527 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.607185 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-kube-api-access-bz6pl" pod "b61e5926-471e-4f07-ae43-c988b094340c" (UID: "b61e5926-471e-4f07-ae43-c988b094340c"). InnerVolumeSpecName "kube-api-access-bz6pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:24.704700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.704669 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.704700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.704697 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz6pl\" (UniqueName: \"kubernetes.io/projected/b61e5926-471e-4f07-ae43-c988b094340c-kube-api-access-bz6pl\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:24.933075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.933037 4625 generic.go:356] "Generic (PLEG): container finished" podID="b61e5926-471e-4f07-ae43-c988b094340c" containerID="4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3" exitCode=0 Nov 03 22:47:24.933277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.933082 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" Nov 03 22:47:24.933277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.933121 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" event={"ID":"b61e5926-471e-4f07-ae43-c988b094340c","Type":"ContainerDied","Data":"4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3"} Nov 03 22:47:24.933277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.933184 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s" event={"ID":"b61e5926-471e-4f07-ae43-c988b094340c","Type":"ContainerDied","Data":"2facbd464da16610e305d4bd9671bd725ad9a5163f880d4b6d311eb229086036"} Nov 03 22:47:24.933277 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.933215 4625 scope.go:122] "RemoveContainer" containerID="4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3" Nov 03 22:47:24.951014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.950982 4625 scope.go:122] "RemoveContainer" containerID="4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3" Nov 03 22:47:24.951530 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:47:24.951508 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3\": container with ID starting with 4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3 not found: ID does not exist" containerID="4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3" Nov 03 22:47:24.951637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.951548 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3"} err="failed to get container status \"4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3\": rpc error: code = NotFound desc = could not find container \"4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3\": container with ID starting with 4e9a52d48b3b9eb767803923931e349012ae017360886ddf44c20759146bc7f3 not found: ID does not exist" Nov 03 22:47:24.990463 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.990384 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s"] Nov 03 22:47:24.994987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:24.994962 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ddx5s"] Nov 03 22:47:26.730414 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:26.730380 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61e5926-471e-4f07-ae43-c988b094340c" path="/var/lib/kubelet/pods/b61e5926-471e-4f07-ae43-c988b094340c/volumes" Nov 03 22:47:27.238740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:27.238233 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn"] Nov 03 22:47:27.251006 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:27.249431 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" containerID="cri-o://a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b" gracePeriod=30 Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.643627 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg"] Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644818 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644840 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644879 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644886 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644907 4625 cpu_manager.go:414] "containerMap: removing container" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644914 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644943 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644951 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644970 4625 cpu_manager.go:414] "containerMap: removing container" podUID="613cb909-ee79-4d35-ab13-e290d8736651" containerName="maas-api" Nov 03 22:47:28.657740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.644977 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613cb909-ee79-4d35-ab13-e290d8736651" containerName="maas-api" Nov 03 22:47:28.677748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.661064 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b65df3-8781-4fc4-be9d-8c1df26fa290" containerName="istio-proxy" Nov 03 22:47:28.677748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.661109 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="613cb909-ee79-4d35-ab13-e290d8736651" containerName="maas-api" Nov 03 22:47:28.677748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.661121 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" Nov 03 22:47:28.677748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.661144 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61e5926-471e-4f07-ae43-c988b094340c" containerName="istio-proxy" Nov 03 22:47:28.677748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.661161 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c183151e-5e97-4008-a434-7997f7cec128" containerName="main" Nov 03 22:47:28.783709 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.782248 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-certs\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.785776 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.785742 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-envoy\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.786026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.786007 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-podinfo\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.791162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.791134 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425c365-f5bc-404a-9d37-4c0066f138dc-istiod-ca-cert\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.792079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.792047 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-credential-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.792320 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.792299 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-token\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.792547 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.792530 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-data\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.796278 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.796256 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.796579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.796557 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfxm\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-kube-api-access-kvfxm\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.823831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.813354 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.834579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.834548 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-d4b6bb-openshift-default-dockercfg-lvgrr\"" Nov 03 22:47:28.898010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.897972 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-certs\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.898739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.898694 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-envoy\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.898970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.898952 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-podinfo\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.899123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.899104 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425c365-f5bc-404a-9d37-4c0066f138dc-istiod-ca-cert\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.899358 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.899343 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-credential-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.899472 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.899459 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-token\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.899625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.899608 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-data\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.899936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.899917 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.901045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.900166 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfxm\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-kube-api-access-kvfxm\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.905186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.901876 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425c365-f5bc-404a-9d37-4c0066f138dc-istiod-ca-cert\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.906773 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.898634 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-certs\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.906915 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.904354 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-credential-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.907032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.904587 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-data\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.907413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.907389 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.941510 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.941369 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-token\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.952561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.952495 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfxm\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-kube-api-access-kvfxm\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.963848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.959307 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-podinfo\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.963848 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.959852 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-envoy\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:28.969906 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:28.969831 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg"] Nov 03 22:47:29.203797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:29.194985 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:29.573738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:29.573699 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg"] Nov 03 22:47:29.576047 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:47:29.576008 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0425c365_f5bc_404a_9d37_4c0066f138dc.slice/crio-d80af6d82016c97afe8b02deabb2eab20aa88b3c8bf04b5313ff93fc0918937a WatchSource:0}: Error finding container d80af6d82016c97afe8b02deabb2eab20aa88b3c8bf04b5313ff93fc0918937a: Status 404 returned error can't find the container with id d80af6d82016c97afe8b02deabb2eab20aa88b3c8bf04b5313ff93fc0918937a Nov 03 22:47:29.579612 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:29.579579 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:47:29.579770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:29.579646 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:47:29.579770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:29.579675 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:47:30.103488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:30.103449 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" event={"ID":"0425c365-f5bc-404a-9d37-4c0066f138dc","Type":"ContainerStarted","Data":"ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e"} Nov 03 22:47:30.103934 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:30.103541 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" event={"ID":"0425c365-f5bc-404a-9d37-4c0066f138dc","Type":"ContainerStarted","Data":"d80af6d82016c97afe8b02deabb2eab20aa88b3c8bf04b5313ff93fc0918937a"} Nov 03 22:47:30.150296 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:30.150233 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podStartSLOduration=2.150217736 podStartE2EDuration="2.150217736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:30.14683073 +0000 UTC m=+2071.576166722" watchObservedRunningTime="2026-11-03 22:47:30.150217736 +0000 UTC m=+2071.579553727" Nov 03 22:47:30.226803 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:30.226770 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:30.232513 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:30.232487 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:31.111761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:31.111710 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:31.112855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:31.112836 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:47:32.189774 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:47:32.189737 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:47:32.666192 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.666167 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:47:32.793300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793183 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-envoy\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793296 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-podinfo\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793399 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-data\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793490 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-token\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793519 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-kube-api-access-ckfdj\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-kube-api-access-ckfdj\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793616 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793556 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-credential-socket\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793634 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istiod-ca-cert\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793754 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-certs\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.793857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.793852 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-socket\") pod \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\" (UID: \"e3ff44b9-2e63-4538-a16f-0bdd609e16d7\") " Nov 03 22:47:32.795360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.794551 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istiod-ca-cert" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:32.795360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.795020 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-socket" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:32.795360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.795237 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-certs" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:32.795360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.795284 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-credential-socket" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:32.795973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.795949 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.796105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.795978 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.796105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.795993 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.796105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.796007 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.796105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.796052 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-envoy" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:32.796372 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.796346 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-data" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:32.798681 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.798656 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-podinfo" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:32.800109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.800082 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-kube-api-access-ckfdj" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "kube-api-access-ckfdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:32.801347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.801316 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-token" pod "e3ff44b9-2e63-4538-a16f-0bdd609e16d7" (UID: "e3ff44b9-2e63-4538-a16f-0bdd609e16d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:32.897645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.897609 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.897645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.897643 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.897645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.897655 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.897915 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.897666 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:32.897915 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:32.897679 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfdj\" (UniqueName: \"kubernetes.io/projected/e3ff44b9-2e63-4538-a16f-0bdd609e16d7-kube-api-access-ckfdj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:47:33.153604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.153571 4625 generic.go:356] "Generic (PLEG): container finished" podID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerID="a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b" exitCode=0 Nov 03 22:47:33.154455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.154428 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" Nov 03 22:47:33.154811 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.154788 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" event={"ID":"e3ff44b9-2e63-4538-a16f-0bdd609e16d7","Type":"ContainerDied","Data":"a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b"} Nov 03 22:47:33.154904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.154842 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn" event={"ID":"e3ff44b9-2e63-4538-a16f-0bdd609e16d7","Type":"ContainerDied","Data":"6968c10cba5ed42446778dcbc0c4511d435adc2b71a656f24a1c9cf4cb04758f"} Nov 03 22:47:33.154904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.154874 4625 scope.go:122] "RemoveContainer" containerID="a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b" Nov 03 22:47:33.171939 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.171909 4625 scope.go:122] "RemoveContainer" containerID="637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9" Nov 03 22:47:33.183223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.183200 4625 scope.go:122] "RemoveContainer" containerID="a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b" Nov 03 22:47:33.183587 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:47:33.183552 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b\": container with ID starting with a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b not found: ID does not exist" containerID="a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b" Nov 03 22:47:33.183686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.183603 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b"} err="failed to get container status \"a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b\": rpc error: code = NotFound desc = could not find container \"a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b\": container with ID starting with a9d0226f59d2191702db7df221583ebc3d2ae61c5443eb79d4d7aa4bf4c4307b not found: ID does not exist" Nov 03 22:47:33.183686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.183624 4625 scope.go:122] "RemoveContainer" containerID="637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9" Nov 03 22:47:33.183948 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:47:33.183926 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9\": container with ID starting with 637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9 not found: ID does not exist" containerID="637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9" Nov 03 22:47:33.184009 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.183955 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9"} err="failed to get container status \"637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9\": rpc error: code = NotFound desc = could not find container \"637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9\": container with ID starting with 637cd577bcb18052b275561d2da4e8a24d57e75b8e5ab2fa90cc4253573b16b9 not found: ID does not exist" Nov 03 22:47:33.214751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.214685 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn"] Nov 03 22:47:33.224084 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:33.224057 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2d6076-openshift-default-5bdb59ccf9-48ljn"] Nov 03 22:47:34.732882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:47:34.732843 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" path="/var/lib/kubelet/pods/e3ff44b9-2e63-4538-a16f-0bdd609e16d7/volumes" Nov 03 22:48:02.190314 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:48:02.190276 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:48:03.750479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:03.750444 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:48:03.777602 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:03.777577 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:48:11.446746 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:11.446540 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn"] Nov 03 22:48:11.448754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:11.446802 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" containerID="cri-o://5b57d163b01b14a9f97954b1e9f582d5369436a0b8cb60b7d9991097fdde504d" gracePeriod=30 Nov 03 22:48:12.024936 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:12.024904 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v"] Nov 03 22:48:16.728370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.728329 4625 generic.go:356] "Generic (PLEG): container finished" podID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerID="5b57d163b01b14a9f97954b1e9f582d5369436a0b8cb60b7d9991097fdde504d" exitCode=0 Nov 03 22:48:16.737777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.737747 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" event={"ID":"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e","Type":"ContainerDied","Data":"5b57d163b01b14a9f97954b1e9f582d5369436a0b8cb60b7d9991097fdde504d"} Nov 03 22:48:16.832096 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.831794 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951210 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-credential-socket\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951273 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-token\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-token\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951331 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-kube-api-access-g2dqw\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-kube-api-access-g2dqw\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951417 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-envoy\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951448 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-podinfo\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951491 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istiod-ca-cert\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951526 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-socket\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951556 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-data\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.951750 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-certs\") pod \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\" (UID: \"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e\") " Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.953097 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-credential-socket" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.958833 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istiod-ca-cert" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.959094 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-socket" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.980305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.959156 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-data" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.998076 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:16.994058 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-envoy" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:17.001036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.000977 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-podinfo" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:48:17.011138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.010985 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-token" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:17.029274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.025822 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-certs" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057462 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057504 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057520 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057533 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057546 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057556 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057566 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.057611 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057577 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.058092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.057771 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-kube-api-access-g2dqw" pod "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" (UID: "3cd03a2f-53f5-43cb-bb38-0d0aed39e22e"). InnerVolumeSpecName "kube-api-access-g2dqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:17.158882 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.158849 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2dqw\" (UniqueName: \"kubernetes.io/projected/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e-kube-api-access-g2dqw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.769173 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.769133 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" event={"ID":"3cd03a2f-53f5-43cb-bb38-0d0aed39e22e","Type":"ContainerDied","Data":"6d157f7087cdd1e276ca1b83343d57a3cd494233673c1ce83c54b5a60beb8bfb"} Nov 03 22:48:17.769815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.769792 4625 scope.go:122] "RemoveContainer" containerID="5b57d163b01b14a9f97954b1e9f582d5369436a0b8cb60b7d9991097fdde504d" Nov 03 22:48:17.770099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.770083 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn" Nov 03 22:48:17.892800 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.892764 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn"] Nov 03 22:48:17.933643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:17.933599 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-kjjgn"] Nov 03 22:48:18.731886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:18.731855 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" path="/var/lib/kubelet/pods/3cd03a2f-53f5-43cb-bb38-0d0aed39e22e/volumes" Nov 03 22:48:22.454364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:22.454325 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd"] Nov 03 22:48:27.053747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.053665 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" podUID="1954d4e8-bf6d-4635-950b-dcd56a6f96f3" containerName="main" containerID="cri-o://e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516" gracePeriod=45 Nov 03 22:48:27.437191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.437153 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:48:27.455833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.455767 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" podUID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" containerName="payload-pre-processing" containerID="cri-o://c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623" gracePeriod=25 Nov 03 22:48:27.489760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.489676 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tls-certs\") pod \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " Nov 03 22:48:27.489941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.489872 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-model-cache\") pod \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " Nov 03 22:48:27.490024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490006 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-kube-api-access-wt9mw\" (UniqueName: \"kubernetes.io/projected/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-kube-api-access-wt9mw\") pod \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " Nov 03 22:48:27.490175 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490148 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-model-cache" pod "1954d4e8-bf6d-4635-950b-dcd56a6f96f3" (UID: "1954d4e8-bf6d-4635-950b-dcd56a6f96f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:27.490175 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490172 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-home\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-home\") pod \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " Nov 03 22:48:27.490361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490214 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tmp-dir\") pod \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " Nov 03 22:48:27.490361 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490250 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-dshm\") pod \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\" (UID: \"1954d4e8-bf6d-4635-950b-dcd56a6f96f3\") " Nov 03 22:48:27.490464 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490406 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-home" pod "1954d4e8-bf6d-4635-950b-dcd56a6f96f3" (UID: "1954d4e8-bf6d-4635-950b-dcd56a6f96f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:27.490530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490510 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tmp-dir" pod "1954d4e8-bf6d-4635-950b-dcd56a6f96f3" (UID: "1954d4e8-bf6d-4635-950b-dcd56a6f96f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:27.490987 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.490962 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:27.491098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.491014 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:27.491098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.491031 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:27.492459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.492428 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-kube-api-access-wt9mw" pod "1954d4e8-bf6d-4635-950b-dcd56a6f96f3" (UID: "1954d4e8-bf6d-4635-950b-dcd56a6f96f3"). InnerVolumeSpecName "kube-api-access-wt9mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:27.492459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.492443 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tls-certs" pod "1954d4e8-bf6d-4635-950b-dcd56a6f96f3" (UID: "1954d4e8-bf6d-4635-950b-dcd56a6f96f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:27.492643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.492624 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-dshm" pod "1954d4e8-bf6d-4635-950b-dcd56a6f96f3" (UID: "1954d4e8-bf6d-4635-950b-dcd56a6f96f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:27.592405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.592372 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:27.592405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.592401 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:27.592592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.592412 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt9mw\" (UniqueName: \"kubernetes.io/projected/1954d4e8-bf6d-4635-950b-dcd56a6f96f3-kube-api-access-wt9mw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:27.858675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.858589 4625 generic.go:356] "Generic (PLEG): container finished" podID="1954d4e8-bf6d-4635-950b-dcd56a6f96f3" containerID="e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516" exitCode=0 Nov 03 22:48:27.858675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.858630 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" Nov 03 22:48:27.858675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.858662 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" event={"ID":"1954d4e8-bf6d-4635-950b-dcd56a6f96f3","Type":"ContainerDied","Data":"e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516"} Nov 03 22:48:27.858992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.858711 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v" event={"ID":"1954d4e8-bf6d-4635-950b-dcd56a6f96f3","Type":"ContainerDied","Data":"3090efe6d22b85bfbbe0028e39ccf73404fcb41efe694a5cc1ac12304f900082"} Nov 03 22:48:27.858992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.858747 4625 scope.go:122] "RemoveContainer" containerID="e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516" Nov 03 22:48:27.884827 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.884801 4625 scope.go:122] "RemoveContainer" containerID="e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516" Nov 03 22:48:27.885249 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:48:27.885227 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516\": container with ID starting with e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516 not found: ID does not exist" containerID="e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516" Nov 03 22:48:27.885339 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.885259 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516"} err="failed to get container status \"e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516\": rpc error: code = NotFound desc = could not find container \"e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516\": container with ID starting with e0e775465281f848dbb71a13454bb595ad6599b3ceb5a018829b4b388ac53516 not found: ID does not exist" Nov 03 22:48:27.915890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.915856 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v"] Nov 03 22:48:27.928379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:27.928343 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9e540d/test-model-9e540d-kserve-66dc6b979c-58p4v"] Nov 03 22:48:28.731396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:28.731362 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1954d4e8-bf6d-4635-950b-dcd56a6f96f3" path="/var/lib/kubelet/pods/1954d4e8-bf6d-4635-950b-dcd56a6f96f3/volumes" Nov 03 22:48:32.187753 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:48:32.187686 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:48:35.807394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:35.807354 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" podUID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:48:45.807353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:45.807312 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" podUID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:48:47.758231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:47.758205 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:48:47.889629 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:47.889593 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-kube-api-access-jfcvp\" (UniqueName: \"kubernetes.io/projected/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-kube-api-access-jfcvp\") pod \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " Nov 03 22:48:47.889962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:47.889946 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-plugins-config-volume\") pod \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\" (UID: \"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff\") " Nov 03 22:48:47.890386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:47.890364 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-plugins-config-volume" pod "0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" (UID: "0fee21b5-5b6d-4403-b1fc-e5e8acf1feff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:47.890596 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:47.890583 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:47.891839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:47.891818 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-kube-api-access-jfcvp" pod "0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" (UID: "0fee21b5-5b6d-4403-b1fc-e5e8acf1feff"). InnerVolumeSpecName "kube-api-access-jfcvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:47.992212 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:47.992133 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcvp\" (UniqueName: \"kubernetes.io/projected/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff-kube-api-access-jfcvp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:48.163872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.163833 4625 generic.go:356] "Generic (PLEG): container finished" podID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" containerID="c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623" exitCode=0 Nov 03 22:48:48.164040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.163882 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" Nov 03 22:48:48.164040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.163913 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" event={"ID":"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff","Type":"ContainerDied","Data":"c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623"} Nov 03 22:48:48.164040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.163982 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd" event={"ID":"0fee21b5-5b6d-4403-b1fc-e5e8acf1feff","Type":"ContainerDied","Data":"034dcaec95c5df87b2274e8143eb1ee680501fbe5866090d9d101d01e7987172"} Nov 03 22:48:48.164040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.164014 4625 scope.go:122] "RemoveContainer" containerID="c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623" Nov 03 22:48:48.180942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.180919 4625 scope.go:122] "RemoveContainer" containerID="c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623" Nov 03 22:48:48.181480 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:48:48.181459 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623\": container with ID starting with c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623 not found: ID does not exist" containerID="c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623" Nov 03 22:48:48.181564 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.181488 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623"} err="failed to get container status \"c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623\": rpc error: code = NotFound desc = could not find container \"c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623\": container with ID starting with c141d8ed3432d94f3d1954156db12c15735034c037300cd37462605caa0aa623 not found: ID does not exist" Nov 03 22:48:48.226501 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.226473 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd"] Nov 03 22:48:48.230640 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.230611 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7065-7bc48cbbf9-psdjd"] Nov 03 22:48:48.798334 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:48.798241 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" path="/var/lib/kubelet/pods/0fee21b5-5b6d-4403-b1fc-e5e8acf1feff/volumes" Nov 03 22:48:54.566676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:54.566640 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp"] Nov 03 22:48:54.567687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:54.567649 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" podUID="72ef9b2a-3bf4-403e-a701-aed993efd758" containerName="maas-api" containerID="cri-o://95429a2e56433c63b1256ee982e3cf3920fbd3991382611dfd16020b54a6a955" gracePeriod=30 Nov 03 22:48:55.305978 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.305691 4625 generic.go:356] "Generic (PLEG): container finished" podID="72ef9b2a-3bf4-403e-a701-aed993efd758" containerID="95429a2e56433c63b1256ee982e3cf3920fbd3991382611dfd16020b54a6a955" exitCode=0 Nov 03 22:48:55.306217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.306176 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" event={"ID":"72ef9b2a-3bf4-403e-a701-aed993efd758","Type":"ContainerDied","Data":"95429a2e56433c63b1256ee982e3cf3920fbd3991382611dfd16020b54a6a955"} Nov 03 22:48:55.306322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.306244 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" event={"ID":"72ef9b2a-3bf4-403e-a701-aed993efd758","Type":"ContainerDied","Data":"5f66bc0e946ac9c240dc530bed1411543ee6492b260233f1cda13b4896e3be2e"} Nov 03 22:48:55.306322 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.306270 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f66bc0e946ac9c240dc530bed1411543ee6492b260233f1cda13b4896e3be2e" Nov 03 22:48:55.310470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.310132 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:48:55.411411 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.411376 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ef9b2a-3bf4-403e-a701-aed993efd758-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef9b2a-3bf4-403e-a701-aed993efd758-openshift-service-ca\") pod \"72ef9b2a-3bf4-403e-a701-aed993efd758\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " Nov 03 22:48:55.411578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.411479 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-maas-api-tls\") pod \"72ef9b2a-3bf4-403e-a701-aed993efd758\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " Nov 03 22:48:55.411578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.411520 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ef9b2a-3bf4-403e-a701-aed993efd758-kube-api-access-jcmrs\" (UniqueName: \"kubernetes.io/projected/72ef9b2a-3bf4-403e-a701-aed993efd758-kube-api-access-jcmrs\") pod \"72ef9b2a-3bf4-403e-a701-aed993efd758\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " Nov 03 22:48:55.411578 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.411573 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-metrics-certs\") pod \"72ef9b2a-3bf4-403e-a701-aed993efd758\" (UID: \"72ef9b2a-3bf4-403e-a701-aed993efd758\") " Nov 03 22:48:55.411877 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.411849 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ef9b2a-3bf4-403e-a701-aed993efd758-openshift-service-ca" pod "72ef9b2a-3bf4-403e-a701-aed993efd758" (UID: "72ef9b2a-3bf4-403e-a701-aed993efd758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:55.412393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.412358 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef9b2a-3bf4-403e-a701-aed993efd758-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:55.413638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.413612 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-maas-api-tls" pod "72ef9b2a-3bf4-403e-a701-aed993efd758" (UID: "72ef9b2a-3bf4-403e-a701-aed993efd758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:55.413823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.413678 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-metrics-certs" pod "72ef9b2a-3bf4-403e-a701-aed993efd758" (UID: "72ef9b2a-3bf4-403e-a701-aed993efd758"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:55.414860 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.414839 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ef9b2a-3bf4-403e-a701-aed993efd758-kube-api-access-jcmrs" pod "72ef9b2a-3bf4-403e-a701-aed993efd758" (UID: "72ef9b2a-3bf4-403e-a701-aed993efd758"). InnerVolumeSpecName "kube-api-access-jcmrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:55.519794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.519122 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcmrs\" (UniqueName: \"kubernetes.io/projected/72ef9b2a-3bf4-403e-a701-aed993efd758-kube-api-access-jcmrs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:55.519794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.519166 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:55.519794 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:55.519181 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72ef9b2a-3bf4-403e-a701-aed993efd758-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:48:56.314315 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:56.314282 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp" Nov 03 22:48:56.373208 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:56.373173 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp"] Nov 03 22:48:56.384844 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:56.384814 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7065-6bdb98d458-p2nbp"] Nov 03 22:48:56.729820 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:48:56.729791 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ef9b2a-3bf4-403e-a701-aed993efd758" path="/var/lib/kubelet/pods/72ef9b2a-3bf4-403e-a701-aed993efd758/volumes" Nov 03 22:49:01.801400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:01.801360 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf"] Nov 03 22:49:01.823548 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:01.823513 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-serving-cert: secret "maas-api-e2e-inf-a-9e540d-serving-cert" not found Nov 03 22:49:01.823763 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:01.823621 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls podName:795a8d6c-f9ff-48ff-bd76-8559836e1aec nodeName:}" failed. No retries permitted until 2026-11-03 22:49:02.323599236 +0000 UTC m=+2163.752935225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls") pod "maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" (UID: "795a8d6c-f9ff-48ff-bd76-8559836e1aec") : secret "maas-api-e2e-inf-a-9e540d-serving-cert" not found Nov 03 22:49:02.392081 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:02.389047 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-serving-cert: secret "maas-api-e2e-inf-a-9e540d-serving-cert" not found Nov 03 22:49:02.392081 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:02.389137 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls podName:795a8d6c-f9ff-48ff-bd76-8559836e1aec nodeName:}" failed. No retries permitted until 2026-11-03 22:49:03.389114805 +0000 UTC m=+2164.818450790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls") pod "maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" (UID: "795a8d6c-f9ff-48ff-bd76-8559836e1aec") : secret "maas-api-e2e-inf-a-9e540d-serving-cert" not found Nov 03 22:49:02.445982 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:02.438898 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" podUID="795a8d6c-f9ff-48ff-bd76-8559836e1aec" containerName="maas-api" containerID="cri-o://c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872" gracePeriod=30 Nov 03 22:49:02.711519 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:02.710351 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:49:02.718994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:02.718189 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw"] Nov 03 22:49:02.719081 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:02.719015 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" containerID="cri-o://56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96" gracePeriod=30 Nov 03 22:49:03.159633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.159600 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:49:03.182726 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.182674 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795a8d6c-f9ff-48ff-bd76-8559836e1aec-kube-api-access-rdq4f\" (UniqueName: \"kubernetes.io/projected/795a8d6c-f9ff-48ff-bd76-8559836e1aec-kube-api-access-rdq4f\") pod \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " Nov 03 22:49:03.182913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.182864 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-metrics-certs\") pod \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " Nov 03 22:49:03.182981 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.182929 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795a8d6c-f9ff-48ff-bd76-8559836e1aec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795a8d6c-f9ff-48ff-bd76-8559836e1aec-openshift-service-ca\") pod \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " Nov 03 22:49:03.183120 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.183100 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls\") pod \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\" (UID: \"795a8d6c-f9ff-48ff-bd76-8559836e1aec\") " Nov 03 22:49:03.183319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.183291 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795a8d6c-f9ff-48ff-bd76-8559836e1aec-openshift-service-ca" pod "795a8d6c-f9ff-48ff-bd76-8559836e1aec" (UID: "795a8d6c-f9ff-48ff-bd76-8559836e1aec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:03.183765 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.183742 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795a8d6c-f9ff-48ff-bd76-8559836e1aec-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:03.185059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.185035 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795a8d6c-f9ff-48ff-bd76-8559836e1aec-kube-api-access-rdq4f" pod "795a8d6c-f9ff-48ff-bd76-8559836e1aec" (UID: "795a8d6c-f9ff-48ff-bd76-8559836e1aec"). InnerVolumeSpecName "kube-api-access-rdq4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:03.185159 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.185137 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls" pod "795a8d6c-f9ff-48ff-bd76-8559836e1aec" (UID: "795a8d6c-f9ff-48ff-bd76-8559836e1aec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:03.186886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.186866 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-metrics-certs" pod "795a8d6c-f9ff-48ff-bd76-8559836e1aec" (UID: "795a8d6c-f9ff-48ff-bd76-8559836e1aec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:03.284885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.284786 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:03.284885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.284833 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdq4f\" (UniqueName: \"kubernetes.io/projected/795a8d6c-f9ff-48ff-bd76-8559836e1aec-kube-api-access-rdq4f\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:03.284885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.284843 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/795a8d6c-f9ff-48ff-bd76-8559836e1aec-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:03.473263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.473227 4625 generic.go:356] "Generic (PLEG): container finished" podID="795a8d6c-f9ff-48ff-bd76-8559836e1aec" containerID="c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872" exitCode=0 Nov 03 22:49:03.473432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.473289 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" Nov 03 22:49:03.473432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.473314 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" event={"ID":"795a8d6c-f9ff-48ff-bd76-8559836e1aec","Type":"ContainerDied","Data":"c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872"} Nov 03 22:49:03.473432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.473392 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf" event={"ID":"795a8d6c-f9ff-48ff-bd76-8559836e1aec","Type":"ContainerDied","Data":"90d6c39fe0d6d968310eaa0eea7087aba6e83d76e10d553646d0f209e24f93fd"} Nov 03 22:49:03.473432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.473416 4625 scope.go:122] "RemoveContainer" containerID="c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872" Nov 03 22:49:03.485833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.485809 4625 scope.go:122] "RemoveContainer" containerID="c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872" Nov 03 22:49:03.486128 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:03.486106 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872\": container with ID starting with c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872 not found: ID does not exist" containerID="c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872" Nov 03 22:49:03.486207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.486140 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872"} err="failed to get container status \"c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872\": rpc error: code = NotFound desc = could not find container \"c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872\": container with ID starting with c7a51c3077696f278d252cc97072049b509b9fa8c6836104dd9c769e081eb872 not found: ID does not exist" Nov 03 22:49:03.526782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.526746 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf"] Nov 03 22:49:03.544810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:03.544702 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e540d-655f6cdfdc-xh4kf"] Nov 03 22:49:04.768759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:04.767805 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795a8d6c-f9ff-48ff-bd76-8559836e1aec" path="/var/lib/kubelet/pods/795a8d6c-f9ff-48ff-bd76-8559836e1aec/volumes" Nov 03 22:49:06.538711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:06.538674 4625 scope.go:122] "RemoveContainer" containerID="95429a2e56433c63b1256ee982e3cf3920fbd3991382611dfd16020b54a6a955" Nov 03 22:49:08.111623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.111595 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:49:08.202797 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.202756 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istiod-ca-cert\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.202874 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-kube-api-access-nmxlr\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-kube-api-access-nmxlr\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.202914 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-credential-socket\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.202961 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-data\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.203068 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-envoy\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.203113 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-podinfo\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.203168 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-socket\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.203234 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-certs\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.203410 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-token\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-token\") pod \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\" (UID: \"d90a1d02-4cca-4c64-8d2d-485a03eac17e\") " Nov 03 22:49:08.203945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.203831 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-data" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:08.204098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.204021 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-socket" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:08.204200 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.204181 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.204266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.204207 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.204325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.204271 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-certs" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:08.204403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.204330 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-credential-socket" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:08.204770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.204733 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istiod-ca-cert" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:08.206913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.206884 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-kube-api-access-nmxlr" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "kube-api-access-nmxlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:08.207005 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.206974 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-envoy" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:08.207503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.207477 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-podinfo" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:08.208391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.208365 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-token" pod "d90a1d02-4cca-4c64-8d2d-485a03eac17e" (UID: "d90a1d02-4cca-4c64-8d2d-485a03eac17e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:08.305456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.305425 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.305456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.305454 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.305456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.305462 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.305697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.305470 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.305697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.305477 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90a1d02-4cca-4c64-8d2d-485a03eac17e-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.305697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.305485 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxlr\" (UniqueName: \"kubernetes.io/projected/d90a1d02-4cca-4c64-8d2d-485a03eac17e-kube-api-access-nmxlr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.305697 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.305493 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90a1d02-4cca-4c64-8d2d-485a03eac17e-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:49:08.527966 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.527932 4625 generic.go:356] "Generic (PLEG): container finished" podID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerID="56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96" exitCode=0 Nov 03 22:49:08.528142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.527977 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" Nov 03 22:49:08.528142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.528019 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" event={"ID":"d90a1d02-4cca-4c64-8d2d-485a03eac17e","Type":"ContainerDied","Data":"56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96"} Nov 03 22:49:08.528142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.528098 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw" event={"ID":"d90a1d02-4cca-4c64-8d2d-485a03eac17e","Type":"ContainerDied","Data":"3c81b3074a831df3eb9533fd05cf7ffdf788de02d3a3fc0902bbc12accf15eb3"} Nov 03 22:49:08.528142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.528119 4625 scope.go:122] "RemoveContainer" containerID="56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96" Nov 03 22:49:08.548400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.548364 4625 scope.go:122] "RemoveContainer" containerID="be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d" Nov 03 22:49:08.559164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.559128 4625 scope.go:122] "RemoveContainer" containerID="56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96" Nov 03 22:49:08.559505 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:08.559476 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96\": container with ID starting with 56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96 not found: ID does not exist" containerID="56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96" Nov 03 22:49:08.559607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.559510 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96"} err="failed to get container status \"56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96\": rpc error: code = NotFound desc = could not find container \"56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96\": container with ID starting with 56ad03cc1b4f480cdc4a44721027c11d4bec75e487fe5934afa151b651baee96 not found: ID does not exist" Nov 03 22:49:08.559607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.559534 4625 scope.go:122] "RemoveContainer" containerID="be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d" Nov 03 22:49:08.559897 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:08.559875 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d\": container with ID starting with be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d not found: ID does not exist" containerID="be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d" Nov 03 22:49:08.559995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.559909 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d"} err="failed to get container status \"be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d\": rpc error: code = NotFound desc = could not find container \"be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d\": container with ID starting with be110b97a09131d52cd9d65ddb913641cf79d4407595c6c26e93f94a33e0bf1d not found: ID does not exist" Nov 03 22:49:08.586486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.586455 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw"] Nov 03 22:49:08.590598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.590561 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-w2wcw"] Nov 03 22:49:08.730372 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:08.730342 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" path="/var/lib/kubelet/pods/d90a1d02-4cca-4c64-8d2d-485a03eac17e/volumes" Nov 03 22:49:32.189109 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:49:32.189064 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:49:42.914531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:42.914496 4625 generic.go:356] "Generic (PLEG): container finished" podID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerID="ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e" exitCode=0 Nov 03 22:49:42.915068 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:42.914563 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" event={"ID":"0425c365-f5bc-404a-9d37-4c0066f138dc","Type":"ContainerDied","Data":"ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e"} Nov 03 22:49:42.915068 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:42.915037 4625 scope.go:122] "RemoveContainer" containerID="ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e" Nov 03 22:49:42.917690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:42.917656 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:49:42.917816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:42.917763 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:49:42.917816 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:42.917807 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:49:43.931531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:43.931495 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" event={"ID":"0425c365-f5bc-404a-9d37-4c0066f138dc","Type":"ContainerStarted","Data":"5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708"} Nov 03 22:49:44.226907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:44.226820 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:49:44.228152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:44.228130 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:49:44.228238 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:44.228184 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 22:49:45.226970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:45.226926 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:49:45.227482 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:45.226978 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 22:49:46.111623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:46.111587 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:49:46.227075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:46.227044 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:49:46.227490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:46.227092 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 22:49:47.227189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:47.227147 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:49:47.227642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:47.227206 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 22:49:48.226366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:48.226329 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:49:48.226552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:48.226389 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 22:49:49.226314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:49.226281 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 22:49:49.226771 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:49.226331 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 22:49:50.231418 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:50.231389 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:49:50.232488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:49:50.232466 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 22:50:00.733899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:00.733864 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp"] Nov 03 22:50:00.734347 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:00.734093 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" containerID="cri-o://9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297" gracePeriod=30 Nov 03 22:50:00.744769 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:00.744741 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv"] Nov 03 22:50:00.745041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:00.745014 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" containerID="cri-o://6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205" gracePeriod=30 Nov 03 22:50:02.439425 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:02.439312 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:50:05.222264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:05.222224 4625 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:05.222755 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:05.222299 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:06.047489 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.047460 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:50:06.084161 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.084133 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:50:06.182377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.182340 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-kube-api-access-ddzbp\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-kube-api-access-ddzbp\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.182654 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.182630 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-kube-api-access-8rwbz\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-kube-api-access-8rwbz\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.182856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.182837 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istiod-ca-cert\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.183024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183008 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-credential-socket\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.183168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183153 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-socket\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.183374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183358 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-envoy\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.183541 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183526 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-token\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-token\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.183633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183223 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istiod-ca-cert" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:06.183698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183451 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-credential-socket" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.183698 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183606 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-socket" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.183970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.183951 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-data\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.184094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184079 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-data\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.184217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184202 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-credential-socket\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.184340 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184325 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-podinfo\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.184508 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184492 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-certs\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.184689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184673 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-envoy\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.184845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184828 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-certs\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.184972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184532 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-data" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.184972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184772 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-credential-socket" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.184972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.184936 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-certs" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.185817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.185403 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-data" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.185817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.185707 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-certs" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.186031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.186012 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-socket\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.186219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.186202 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-podinfo\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.186364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.186348 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15554025-03d4-45b2-82c0-a0aa6ade4e44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15554025-03d4-45b2-82c0-a0aa6ade4e44-istiod-ca-cert\") pod \"15554025-03d4-45b2-82c0-a0aa6ade4e44\" (UID: \"15554025-03d4-45b2-82c0-a0aa6ade4e44\") " Nov 03 22:50:06.186481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.186467 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-token\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-token\") pod \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\" (UID: \"178565ed-ea9b-4f39-8cfc-c6b0415bacf2\") " Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.186264 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-socket" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187302 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187320 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187334 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187348 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187360 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187372 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187384 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187396 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187257 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-podinfo" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:06.187763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.187709 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15554025-03d4-45b2-82c0-a0aa6ade4e44-istiod-ca-cert" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:06.188576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.188540 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-token" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:06.188838 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.188796 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-podinfo" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:06.189376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.189334 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-token" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:06.190489 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.190458 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-envoy" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.196210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.196178 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-kube-api-access-ddzbp" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "kube-api-access-ddzbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:06.196807 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.196602 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-envoy" pod "178565ed-ea9b-4f39-8cfc-c6b0415bacf2" (UID: "178565ed-ea9b-4f39-8cfc-c6b0415bacf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.197920 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.197888 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-kube-api-access-8rwbz" pod "15554025-03d4-45b2-82c0-a0aa6ade4e44" (UID: "15554025-03d4-45b2-82c0-a0aa6ade4e44"). InnerVolumeSpecName "kube-api-access-8rwbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:06.289064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.288967 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289004 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289018 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289030 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289041 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15554025-03d4-45b2-82c0-a0aa6ade4e44-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289052 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15554025-03d4-45b2-82c0-a0aa6ade4e44-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289064 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289065 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15554025-03d4-45b2-82c0-a0aa6ade4e44-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289076 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289088 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddzbp\" (UniqueName: \"kubernetes.io/projected/178565ed-ea9b-4f39-8cfc-c6b0415bacf2-kube-api-access-ddzbp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.289624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.289099 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwbz\" (UniqueName: \"kubernetes.io/projected/15554025-03d4-45b2-82c0-a0aa6ade4e44-kube-api-access-8rwbz\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.310085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.310053 4625 generic.go:356] "Generic (PLEG): container finished" podID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerID="6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205" exitCode=0 Nov 03 22:50:06.310267 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.310111 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" Nov 03 22:50:06.310267 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.310127 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" event={"ID":"178565ed-ea9b-4f39-8cfc-c6b0415bacf2","Type":"ContainerDied","Data":"6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205"} Nov 03 22:50:06.310267 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.310216 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv" event={"ID":"178565ed-ea9b-4f39-8cfc-c6b0415bacf2","Type":"ContainerDied","Data":"2a7a5371ddaf99a95dc7756b14d5842c5a1703303f01e06dff8fec41a2d454f8"} Nov 03 22:50:06.310267 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.310248 4625 scope.go:122] "RemoveContainer" containerID="6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205" Nov 03 22:50:06.312337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.312310 4625 generic.go:356] "Generic (PLEG): container finished" podID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerID="9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297" exitCode=0 Nov 03 22:50:06.312487 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.312392 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" Nov 03 22:50:06.312487 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.312407 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" event={"ID":"15554025-03d4-45b2-82c0-a0aa6ade4e44","Type":"ContainerDied","Data":"9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297"} Nov 03 22:50:06.312487 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.312466 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp" event={"ID":"15554025-03d4-45b2-82c0-a0aa6ade4e44","Type":"ContainerDied","Data":"f91f5af05c0dd7f7379beded25f9bf67e733d6aa08a9dd0b6bc598cfaa469dbb"} Nov 03 22:50:06.323620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.323600 4625 scope.go:122] "RemoveContainer" containerID="6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205" Nov 03 22:50:06.323960 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:06.323941 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205\": container with ID starting with 6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205 not found: ID does not exist" containerID="6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205" Nov 03 22:50:06.324067 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.323970 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205"} err="failed to get container status \"6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205\": rpc error: code = NotFound desc = could not find container \"6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205\": container with ID starting with 6479c0bf9922791b6176ef2c72bfb463c924d6f7c42681cc7546a4882e341205 not found: ID does not exist" Nov 03 22:50:06.324067 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.323991 4625 scope.go:122] "RemoveContainer" containerID="9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297" Nov 03 22:50:06.335105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.335087 4625 scope.go:122] "RemoveContainer" containerID="b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1" Nov 03 22:50:06.345693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.345670 4625 scope.go:122] "RemoveContainer" containerID="9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297" Nov 03 22:50:06.346094 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:06.346073 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297\": container with ID starting with 9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297 not found: ID does not exist" containerID="9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297" Nov 03 22:50:06.346155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.346102 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297"} err="failed to get container status \"9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297\": rpc error: code = NotFound desc = could not find container \"9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297\": container with ID starting with 9e4e7d921b01b6c74015c8df156f1f47c46086582f7924a2da0f270b015fa297 not found: ID does not exist" Nov 03 22:50:06.346155 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.346123 4625 scope.go:122] "RemoveContainer" containerID="b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1" Nov 03 22:50:06.346398 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:06.346380 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1\": container with ID starting with b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1 not found: ID does not exist" containerID="b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1" Nov 03 22:50:06.346441 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.346405 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1"} err="failed to get container status \"b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1\": rpc error: code = NotFound desc = could not find container \"b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1\": container with ID starting with b2f7b80b9143417435c7c681ba5dd8d6fb36aa75f8996697d6d5ab04d39b49b1 not found: ID does not exist" Nov 03 22:50:06.370301 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.370267 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp"] Nov 03 22:50:06.374958 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.374931 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-zqkwp"] Nov 03 22:50:06.408060 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.408024 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv"] Nov 03 22:50:06.412750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.412695 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-bgwbv"] Nov 03 22:50:06.797862 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.797831 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" path="/var/lib/kubelet/pods/15554025-03d4-45b2-82c0-a0aa6ade4e44/volumes" Nov 03 22:50:06.798402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:06.798388 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" path="/var/lib/kubelet/pods/178565ed-ea9b-4f39-8cfc-c6b0415bacf2/volumes" Nov 03 22:50:18.290460 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:18.290427 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9"] Nov 03 22:50:18.290888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:18.290668 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" containerID="cri-o://a83b3eee866a98867879cd0d2ac5261fd3025ee138191d39a2d11bb2f516e7fb" gracePeriod=30 Nov 03 22:50:19.755105 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:19.755070 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f"] Nov 03 22:50:19.755712 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:19.755276 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" containerID="cri-o://4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd" gracePeriod=30 Nov 03 22:50:19.769113 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:19.769081 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p"] Nov 03 22:50:19.769690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:19.769351 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" containerID="cri-o://aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811" gracePeriod=30 Nov 03 22:50:22.267945 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:22.267892 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-5w59f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:22.268629 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:22.268514 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:23.462639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.462604 4625 generic.go:356] "Generic (PLEG): container finished" podID="2af5050e-5e51-4d60-a172-64c2d154a063" containerID="a83b3eee866a98867879cd0d2ac5261fd3025ee138191d39a2d11bb2f516e7fb" exitCode=0 Nov 03 22:50:23.463094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.462675 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" event={"ID":"2af5050e-5e51-4d60-a172-64c2d154a063","Type":"ContainerDied","Data":"a83b3eee866a98867879cd0d2ac5261fd3025ee138191d39a2d11bb2f516e7fb"} Nov 03 22:50:23.463094 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.462754 4625 scope.go:122] "RemoveContainer" containerID="659a948bb9b9529f05c14af772ecb97f1f721a452308f4463928d83ea49b7abe" Nov 03 22:50:23.590498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.590470 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:50:23.764830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.764796 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-credential-socket\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.764894 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-data\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765000 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.764990 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2af5050e-5e51-4d60-a172-64c2d154a063-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af5050e-5e51-4d60-a172-64c2d154a063-istio-podinfo\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765060 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-istio-token\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-istio-token\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765082 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-credential-socket" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.765226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765113 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-data" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.765226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765131 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-kube-api-access-jpxn4\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-kube-api-access-jpxn4\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765338 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765227 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-envoy\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765377 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-certs\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765406 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-socket\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.765458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765434 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af5050e-5e51-4d60-a172-64c2d154a063-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af5050e-5e51-4d60-a172-64c2d154a063-istiod-ca-cert\") pod \"2af5050e-5e51-4d60-a172-64c2d154a063\" (UID: \"2af5050e-5e51-4d60-a172-64c2d154a063\") " Nov 03 22:50:23.766083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765702 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-certs" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.766083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.765813 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-socket" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.766083 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.766044 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af5050e-5e51-4d60-a172-64c2d154a063-istiod-ca-cert" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:23.766431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.766214 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.766431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.766234 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.766431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.766247 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2af5050e-5e51-4d60-a172-64c2d154a063-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.766431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.766261 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.766431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.766274 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.767417 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.767392 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2af5050e-5e51-4d60-a172-64c2d154a063-istio-podinfo" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:23.767565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.767549 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-istio-token" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:23.767911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.767884 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-kube-api-access-jpxn4" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "kube-api-access-jpxn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:23.768353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.768326 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-envoy" pod "2af5050e-5e51-4d60-a172-64c2d154a063" (UID: "2af5050e-5e51-4d60-a172-64c2d154a063"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.868737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.868146 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2af5050e-5e51-4d60-a172-64c2d154a063-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.868737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.868182 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.868737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.868195 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpxn4\" (UniqueName: \"kubernetes.io/projected/2af5050e-5e51-4d60-a172-64c2d154a063-kube-api-access-jpxn4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.868737 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:23.868209 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2af5050e-5e51-4d60-a172-64c2d154a063-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:24.473857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:24.473814 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" Nov 03 22:50:24.473857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:24.473849 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9" event={"ID":"2af5050e-5e51-4d60-a172-64c2d154a063","Type":"ContainerDied","Data":"39858a830aff83abb37bb59d82a452096af646ad34563c593d4295d14ff41958"} Nov 03 22:50:24.474379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:24.473925 4625 scope.go:122] "RemoveContainer" containerID="a83b3eee866a98867879cd0d2ac5261fd3025ee138191d39a2d11bb2f516e7fb" Nov 03 22:50:24.543378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:24.543341 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9"] Nov 03 22:50:24.553307 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:24.553279 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-td5c9"] Nov 03 22:50:24.729171 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:24.729099 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" path="/var/lib/kubelet/pods/2af5050e-5e51-4d60-a172-64c2d154a063/volumes" Nov 03 22:50:25.076839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.076810 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:50:25.107004 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.106969 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:50:25.185678 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.185644 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-socket\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.185886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.185738 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-token\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-token\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.185886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.185815 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istiod-ca-cert\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.185886 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.185854 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-kube-api-access-hrwtj\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-kube-api-access-hrwtj\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.186012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.185892 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-envoy\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.186012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.185957 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-socket" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.186012 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.185974 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-podinfo\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.186112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186044 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-credential-socket\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.186112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186100 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-data\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.186222 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186205 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istiod-ca-cert" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:25.186260 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186208 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-certs\") pod \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\" (UID: \"e1dfde9f-d8e4-4e06-928c-559ecdae76e9\") " Nov 03 22:50:25.186462 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186441 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-certs" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.186793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186770 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.186793 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186792 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.186958 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.186803 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.187063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.187041 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-data" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.187354 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.187329 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-credential-socket" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.189598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.189572 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-token" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:25.189598 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.189588 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-podinfo" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:25.189792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.189637 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-envoy" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.189902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.189870 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-kube-api-access-hrwtj" pod "e1dfde9f-d8e4-4e06-928c-559ecdae76e9" (UID: "e1dfde9f-d8e4-4e06-928c-559ecdae76e9"). InnerVolumeSpecName "kube-api-access-hrwtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.307783 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-podinfo\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.307904 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-kube-api-access-nlzr4\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-kube-api-access-nlzr4\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.308012 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istiod-ca-cert\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.308066 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-data\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.308215 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-socket\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.308267 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-certs\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.308292 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-token\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-token\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.308336 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-envoy\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.308368 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-credential-socket\") pod \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\" (UID: \"d82444d7-3c98-4ed4-8ee9-3e227757dbed\") " Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.309049 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.309074 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.309089 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwtj\" (UniqueName: \"kubernetes.io/projected/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-kube-api-access-hrwtj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.309103 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.309116 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.309128 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dfde9f-d8e4-4e06-928c-559ecdae76e9-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.312795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.309397 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-credential-socket" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.315246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.314848 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-socket" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.317691 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.317641 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-certs" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.317971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.317937 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-data" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.318422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.318397 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istiod-ca-cert" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:25.330652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.330592 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-podinfo" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:25.336752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.330993 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-kube-api-access-nlzr4" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "kube-api-access-nlzr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:25.337190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.337123 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-envoy" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:25.337504 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.337472 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-token" pod "d82444d7-3c98-4ed4-8ee9-3e227757dbed" (UID: "d82444d7-3c98-4ed4-8ee9-3e227757dbed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:25.410051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410016 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410045 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410054 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410063 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410327 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410071 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410327 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410078 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82444d7-3c98-4ed4-8ee9-3e227757dbed-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410327 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410087 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410327 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410096 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzr4\" (UniqueName: \"kubernetes.io/projected/d82444d7-3c98-4ed4-8ee9-3e227757dbed-kube-api-access-nlzr4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.410327 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.410104 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82444d7-3c98-4ed4-8ee9-3e227757dbed-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:50:25.483661 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.483625 4625 generic.go:356] "Generic (PLEG): container finished" podID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerID="4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd" exitCode=0 Nov 03 22:50:25.484213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.483676 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" Nov 03 22:50:25.484213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.483744 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" event={"ID":"e1dfde9f-d8e4-4e06-928c-559ecdae76e9","Type":"ContainerDied","Data":"4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd"} Nov 03 22:50:25.484213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.483813 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f" event={"ID":"e1dfde9f-d8e4-4e06-928c-559ecdae76e9","Type":"ContainerDied","Data":"5f0d9bc43671a80d705d9c171375ff8b8da4a787a045d2c2c31926405f7e0817"} Nov 03 22:50:25.484213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.483846 4625 scope.go:122] "RemoveContainer" containerID="4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd" Nov 03 22:50:25.487217 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.487185 4625 generic.go:356] "Generic (PLEG): container finished" podID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerID="aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811" exitCode=0 Nov 03 22:50:25.487323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.487218 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" event={"ID":"d82444d7-3c98-4ed4-8ee9-3e227757dbed","Type":"ContainerDied","Data":"aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811"} Nov 03 22:50:25.487323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.487244 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" Nov 03 22:50:25.487323 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.487253 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p" event={"ID":"d82444d7-3c98-4ed4-8ee9-3e227757dbed","Type":"ContainerDied","Data":"77f803923557efc7787fa6a0fbac3ed0d3aa71029d0684cc0d05dd0fa5c77edd"} Nov 03 22:50:25.495941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.495921 4625 scope.go:122] "RemoveContainer" containerID="4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd" Nov 03 22:50:25.496234 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:25.496215 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd\": container with ID starting with 4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd not found: ID does not exist" containerID="4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd" Nov 03 22:50:25.496325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.496244 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd"} err="failed to get container status \"4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd\": rpc error: code = NotFound desc = could not find container \"4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd\": container with ID starting with 4a3731bd0edb705e23ff8ba817a9952d84ad1f356c20f25f180415d06ea7dcdd not found: ID does not exist" Nov 03 22:50:25.496325 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.496265 4625 scope.go:122] "RemoveContainer" containerID="aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811" Nov 03 22:50:25.506164 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.506144 4625 scope.go:122] "RemoveContainer" containerID="821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7" Nov 03 22:50:25.515566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.515548 4625 scope.go:122] "RemoveContainer" containerID="aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811" Nov 03 22:50:25.515990 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:25.515972 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811\": container with ID starting with aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811 not found: ID does not exist" containerID="aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811" Nov 03 22:50:25.516063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.515996 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811"} err="failed to get container status \"aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811\": rpc error: code = NotFound desc = could not find container \"aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811\": container with ID starting with aa8b19f7a2c010c3a0411cfaa86fb962f11fe38118ec260abf24bae472b8e811 not found: ID does not exist" Nov 03 22:50:25.516063 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.516013 4625 scope.go:122] "RemoveContainer" containerID="821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7" Nov 03 22:50:25.516276 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:25.516254 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7\": container with ID starting with 821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7 not found: ID does not exist" containerID="821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7" Nov 03 22:50:25.516333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.516285 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7"} err="failed to get container status \"821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7\": rpc error: code = NotFound desc = could not find container \"821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7\": container with ID starting with 821476558b22dbda1314ed6ca94a36a487836ec9bbfa392b8facc9ab16f973a7 not found: ID does not exist" Nov 03 22:50:25.538888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.538868 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p"] Nov 03 22:50:25.558856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.558822 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v828p"] Nov 03 22:50:25.586831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.586748 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f"] Nov 03 22:50:25.591444 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:25.591414 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-5w59f"] Nov 03 22:50:26.728602 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:26.728571 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" path="/var/lib/kubelet/pods/d82444d7-3c98-4ed4-8ee9-3e227757dbed/volumes" Nov 03 22:50:26.729182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:26.729163 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" path="/var/lib/kubelet/pods/e1dfde9f-d8e4-4e06-928c-559ecdae76e9/volumes" Nov 03 22:50:32.298032 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:50:32.297987 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:50:39.136270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.136230 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q"] Nov 03 22:50:39.137324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137298 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" Nov 03 22:50:39.137324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137324 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137351 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137359 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137370 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137378 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137413 4625 cpu_manager.go:414] "containerMap: removing container" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137422 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137435 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137443 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137455 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137463 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137475 4625 cpu_manager.go:414] "containerMap: removing container" podUID="795a8d6c-f9ff-48ff-bd76-8559836e1aec" containerName="maas-api" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137482 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795a8d6c-f9ff-48ff-bd76-8559836e1aec" containerName="maas-api" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137493 4625 cpu_manager.go:414] "containerMap: removing container" podUID="72ef9b2a-3bf4-403e-a701-aed993efd758" containerName="maas-api" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137500 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ef9b2a-3bf4-403e-a701-aed993efd758" containerName="maas-api" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137520 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137527 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137543 4625 cpu_manager.go:414] "containerMap: removing container" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" Nov 03 22:50:39.137546 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137550 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137570 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137577 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137587 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137594 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137605 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137613 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137621 4625 cpu_manager.go:414] "containerMap: removing container" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137628 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137654 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1954d4e8-bf6d-4635-950b-dcd56a6f96f3" containerName="main" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137660 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1954d4e8-bf6d-4635-950b-dcd56a6f96f3" containerName="main" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137680 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" containerName="payload-pre-processing" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137687 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" containerName="payload-pre-processing" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137867 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137885 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1dfde9f-d8e4-4e06-928c-559ecdae76e9" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137892 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137903 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="795a8d6c-f9ff-48ff-bd76-8559836e1aec" containerName="maas-api" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137912 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ef9b2a-3bf4-403e-a701-aed993efd758" containerName="maas-api" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137922 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137932 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="178565ed-ea9b-4f39-8cfc-c6b0415bacf2" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137941 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137949 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ff44b9-2e63-4538-a16f-0bdd609e16d7" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137959 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af5050e-5e51-4d60-a172-64c2d154a063" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137968 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1954d4e8-bf6d-4635-950b-dcd56a6f96f3" containerName="main" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137977 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fee21b5-5b6d-4403-b1fc-e5e8acf1feff" containerName="payload-pre-processing" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137988 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.137998 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd03a2f-53f5-43cb-bb38-0d0aed39e22e" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.138009 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.138016 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82444d7-3c98-4ed4-8ee9-3e227757dbed" containerName="istio-proxy" Nov 03 22:50:39.138518 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.138024 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb7bc4c-db84-4ea3-b6dd-75e2d983cd43" containerName="istio-proxy" Nov 03 22:50:39.144390 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.144357 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.161553 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.161523 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q"] Nov 03 22:50:39.166107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.166060 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-d4b6bb\"/\"openshift-service-ca.crt\"" Nov 03 22:50:39.166244 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.166122 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-d4b6bb\"/\"auth-test-model-d4b6bb-kserve-self-signed-certs\"" Nov 03 22:50:39.166391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.166369 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-d4b6bb\"/\"kube-root-ca.crt\"" Nov 03 22:50:39.170625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.170605 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-d4b6bb\"/\"default-dockercfg-9znwn\"" Nov 03 22:50:39.184774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.183155 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-tmp-dir\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.184774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.183242 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8zg\" (UniqueName: \"kubernetes.io/projected/6076e4a0-c1c0-4691-b20c-0a163396385b-kube-api-access-nw8zg\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.184774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.183424 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6076e4a0-c1c0-4691-b20c-0a163396385b-tls-certs\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.184774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.183480 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-dshm\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.184774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.183615 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-model-cache\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.184774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.183672 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-home\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.284524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.284487 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-dshm\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.284694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.284548 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-model-cache\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.284694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.284583 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-home\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.284694 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.284684 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-tmp-dir\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.284855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.284759 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8zg\" (UniqueName: \"kubernetes.io/projected/6076e4a0-c1c0-4691-b20c-0a163396385b-kube-api-access-nw8zg\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.284915 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.284875 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6076e4a0-c1c0-4691-b20c-0a163396385b-tls-certs\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.285032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.285002 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-model-cache\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.285032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.285024 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-home\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.285190 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.285065 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-tmp-dir\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.287039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.287019 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-dshm\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.287374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.287351 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6076e4a0-c1c0-4691-b20c-0a163396385b-tls-certs\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.312693 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.312668 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8zg\" (UniqueName: \"kubernetes.io/projected/6076e4a0-c1c0-4691-b20c-0a163396385b-kube-api-access-nw8zg\") pod \"auth-test-model-d4b6bb-kserve-6499d64885-6x92q\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.469787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.467840 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:39.798916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:39.798886 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q"] Nov 03 22:50:39.800287 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:50:39.800261 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6076e4a0_c1c0_4691_b20c_0a163396385b.slice/crio-45e4de727d5cb863604a2083c274cddb6f90db5df98cf11d60334f7eca3bd535 WatchSource:0}: Error finding container 45e4de727d5cb863604a2083c274cddb6f90db5df98cf11d60334f7eca3bd535: Status 404 returned error can't find the container with id 45e4de727d5cb863604a2083c274cddb6f90db5df98cf11d60334f7eca3bd535 Nov 03 22:50:40.639193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:40.639155 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" event={"ID":"6076e4a0-c1c0-4691-b20c-0a163396385b","Type":"ContainerStarted","Data":"ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198"} Nov 03 22:50:40.639670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:40.639202 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" event={"ID":"6076e4a0-c1c0-4691-b20c-0a163396385b","Type":"ContainerStarted","Data":"45e4de727d5cb863604a2083c274cddb6f90db5df98cf11d60334f7eca3bd535"} Nov 03 22:50:40.681488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:40.681438 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" podStartSLOduration=1.681426234 podStartE2EDuration="1.681426234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:40.679422349 +0000 UTC m=+2262.108758341" watchObservedRunningTime="2026-11-03 22:50:40.681426234 +0000 UTC m=+2262.110762238" Nov 03 22:50:41.468492 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:41.468462 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:46.263807 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.263776 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp"] Nov 03 22:50:46.264490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.264475 4625 cpu_manager.go:414] "containerMap: removing container" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" Nov 03 22:50:46.264550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.264489 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" Nov 03 22:50:46.264550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.264507 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" Nov 03 22:50:46.264550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.264512 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90a1d02-4cca-4c64-8d2d-485a03eac17e" containerName="istio-proxy" Nov 03 22:50:46.264649 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.264634 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="15554025-03d4-45b2-82c0-a0aa6ade4e44" containerName="istio-proxy" Nov 03 22:50:46.268751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.268728 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.289499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.289468 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-b023a2-serving-cert\"" Nov 03 22:50:46.291229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.291200 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp"] Nov 03 22:50:46.415861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.415821 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9jd\" (UniqueName: \"kubernetes.io/projected/18fe9477-aa04-483c-9d15-39df7f73c174-kube-api-access-mn9jd\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.416066 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.415953 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fe9477-aa04-483c-9d15-39df7f73c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.416136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.416119 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.416191 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.416170 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.517344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.517261 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9jd\" (UniqueName: \"kubernetes.io/projected/18fe9477-aa04-483c-9d15-39df7f73c174-kube-api-access-mn9jd\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.517505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.517357 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fe9477-aa04-483c-9d15-39df7f73c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.517552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.517540 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.517624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.517606 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.518342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.518312 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fe9477-aa04-483c-9d15-39df7f73c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.520269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.520238 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.520406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.520388 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.540857 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.540834 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9jd\" (UniqueName: \"kubernetes.io/projected/18fe9477-aa04-483c-9d15-39df7f73c174-kube-api-access-mn9jd\") pod \"maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.594552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.594511 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:46.911341 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.911307 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp"] Nov 03 22:50:46.912682 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:50:46.912652 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fe9477_aa04_483c_9d15_39df7f73c174.slice/crio-4908fd62ef41a98078d519afd2f49dca00a267e894e03baa3538a507d8838e16 WatchSource:0}: Error finding container 4908fd62ef41a98078d519afd2f49dca00a267e894e03baa3538a507d8838e16: Status 404 returned error can't find the container with id 4908fd62ef41a98078d519afd2f49dca00a267e894e03baa3538a507d8838e16 Nov 03 22:50:46.914390 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:46.914369 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:50:47.709138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:47.709038 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" event={"ID":"18fe9477-aa04-483c-9d15-39df7f73c174","Type":"ContainerStarted","Data":"b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050"} Nov 03 22:50:47.709138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:47.709118 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" event={"ID":"18fe9477-aa04-483c-9d15-39df7f73c174","Type":"ContainerStarted","Data":"4908fd62ef41a98078d519afd2f49dca00a267e894e03baa3538a507d8838e16"} Nov 03 22:50:47.709688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:47.709165 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:50:47.755778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:47.755703 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" podStartSLOduration=1.304335268 podStartE2EDuration="1.755685666s" totalImagesPullingTime="451.350398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:47.749536327 +0000 UTC m=+2269.178872319" watchObservedRunningTime="2026-11-03 22:50:47.755685666 +0000 UTC m=+2269.185021660" Nov 03 22:50:49.469778 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:49.469743 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:49.482563 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:49.482536 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:49.735597 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:49.735516 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:50:53.722413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:50:53.722385 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 22:51:02.189310 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:51:02.189273 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:51:32.190083 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:51:32.189999 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:51:33.891822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:33.891782 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n"] Nov 03 22:51:33.897475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:33.897427 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:33.919689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:33.919662 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-9b841d-openshift-default-dockercfg-s842q\"" Nov 03 22:51:33.936092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:33.936063 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n"] Nov 03 22:51:34.006071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006037 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-certs\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006254 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006086 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-data\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006242 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-credential-socket\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006305 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006294 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-socket\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006387 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006363 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-token\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006435 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006397 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-envoy\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006512 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpnq\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-kube-api-access-4lpnq\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006634 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006620 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f93335b-e263-455b-a2eb-d34acc1f1398-istiod-ca-cert\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.006813 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.006796 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-podinfo\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116273 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-podinfo\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116398 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-certs\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116435 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-data\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116525 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-credential-socket\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116564 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-socket\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116601 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-token\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116675 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-envoy\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116772 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpnq\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-kube-api-access-4lpnq\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.116873 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f93335b-e263-455b-a2eb-d34acc1f1398-istiod-ca-cert\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.117742 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f93335b-e263-455b-a2eb-d34acc1f1398-istiod-ca-cert\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.118133 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-certs\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.118388 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-data\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.118680 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-credential-socket\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.122740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.118948 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-socket\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.143351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.143260 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-podinfo\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.150621 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.150555 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpnq\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-kube-api-access-4lpnq\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.154645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.154616 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-token\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.155189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.155128 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-envoy\") pod \"e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.222829 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.222794 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:34.560226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.560144 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n"] Nov 03 22:51:34.564074 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:51:34.564047 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f93335b_e263_455b_a2eb_d34acc1f1398.slice/crio-2e236b4a517cbfbe620c82b6485fc6ab2a14d61a708ddf0502f103225de61ed3 WatchSource:0}: Error finding container 2e236b4a517cbfbe620c82b6485fc6ab2a14d61a708ddf0502f103225de61ed3: Status 404 returned error can't find the container with id 2e236b4a517cbfbe620c82b6485fc6ab2a14d61a708ddf0502f103225de61ed3 Nov 03 22:51:34.567526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.567479 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:51:34.567620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.567568 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:51:34.567620 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:34.567612 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:51:35.204206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:35.204163 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" event={"ID":"3f93335b-e263-455b-a2eb-d34acc1f1398","Type":"ContainerStarted","Data":"387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209"} Nov 03 22:51:35.205016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:35.204230 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" event={"ID":"3f93335b-e263-455b-a2eb-d34acc1f1398","Type":"ContainerStarted","Data":"2e236b4a517cbfbe620c82b6485fc6ab2a14d61a708ddf0502f103225de61ed3"} Nov 03 22:51:35.223815 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:35.223784 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:35.231516 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:35.231490 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:35.260969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:35.260909 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podStartSLOduration=2.26089037 podStartE2EDuration="2.26089037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:35.260513002 +0000 UTC m=+2316.689848991" watchObservedRunningTime="2026-11-03 22:51:35.26089037 +0000 UTC m=+2316.690226365" Nov 03 22:51:36.213639 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:36.213603 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:36.214534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:36.214513 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:51:37.953805 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:37.953769 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl"] Nov 03 22:51:42.955655 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:42.955593 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" podUID="ea0e1694-9167-4b2d-9737-0283eca834e2" containerName="payload-processing" containerID="cri-o://8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b" gracePeriod=25 Nov 03 22:51:43.311433 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.311012 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:51:43.321958 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.321924 4625 generic.go:356] "Generic (PLEG): container finished" podID="ea0e1694-9167-4b2d-9737-0283eca834e2" containerID="8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b" exitCode=0 Nov 03 22:51:43.322145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.322028 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" event={"ID":"ea0e1694-9167-4b2d-9737-0283eca834e2","Type":"ContainerDied","Data":"8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b"} Nov 03 22:51:43.322145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.322083 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" event={"ID":"ea0e1694-9167-4b2d-9737-0283eca834e2","Type":"ContainerDied","Data":"828e65e97ddd9ecac5d3d66fb77203cd661f4e728b65025f12ca766540ec0f74"} Nov 03 22:51:43.322145 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.322115 4625 scope.go:122] "RemoveContainer" containerID="8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b" Nov 03 22:51:43.322293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.322279 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl" Nov 03 22:51:43.338566 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.338241 4625 scope.go:122] "RemoveContainer" containerID="8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b" Nov 03 22:51:43.338778 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:51:43.338711 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b\": container with ID starting with 8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b not found: ID does not exist" containerID="8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b" Nov 03 22:51:43.338873 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.338792 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b"} err="failed to get container status \"8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b\": rpc error: code = NotFound desc = could not find container \"8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b\": container with ID starting with 8bfe9b5957afb4cbf909eae25fa4f38ca7269703e5a720f9ef2ec42087d24e9b not found: ID does not exist" Nov 03 22:51:43.354455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.354407 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0e1694-9167-4b2d-9737-0283eca834e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e1694-9167-4b2d-9737-0283eca834e2-plugins-config-volume\") pod \"ea0e1694-9167-4b2d-9737-0283eca834e2\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " Nov 03 22:51:43.354646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.354598 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0e1694-9167-4b2d-9737-0283eca834e2-kube-api-access-8pl65\" (UniqueName: \"kubernetes.io/projected/ea0e1694-9167-4b2d-9737-0283eca834e2-kube-api-access-8pl65\") pod \"ea0e1694-9167-4b2d-9737-0283eca834e2\" (UID: \"ea0e1694-9167-4b2d-9737-0283eca834e2\") " Nov 03 22:51:43.356056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.356025 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0e1694-9167-4b2d-9737-0283eca834e2-plugins-config-volume" pod "ea0e1694-9167-4b2d-9737-0283eca834e2" (UID: "ea0e1694-9167-4b2d-9737-0283eca834e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:43.358193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.358159 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0e1694-9167-4b2d-9737-0283eca834e2-kube-api-access-8pl65" pod "ea0e1694-9167-4b2d-9737-0283eca834e2" (UID: "ea0e1694-9167-4b2d-9737-0283eca834e2"). InnerVolumeSpecName "kube-api-access-8pl65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:43.456890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.456853 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e1694-9167-4b2d-9737-0283eca834e2-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:51:43.456890 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.456883 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pl65\" (UniqueName: \"kubernetes.io/projected/ea0e1694-9167-4b2d-9737-0283eca834e2-kube-api-access-8pl65\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:51:43.671954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.671916 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl"] Nov 03 22:51:43.683554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:43.683525 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7940ea-56b89d4b64-pvwxl"] Nov 03 22:51:44.759576 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:44.759534 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0e1694-9167-4b2d-9737-0283eca834e2" path="/var/lib/kubelet/pods/ea0e1694-9167-4b2d-9737-0283eca834e2/volumes" Nov 03 22:51:46.523429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.523394 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r"] Nov 03 22:51:46.524531 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.524509 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ea0e1694-9167-4b2d-9737-0283eca834e2" containerName="payload-processing" Nov 03 22:51:46.524604 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.524534 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0e1694-9167-4b2d-9737-0283eca834e2" containerName="payload-processing" Nov 03 22:51:46.524767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.524751 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0e1694-9167-4b2d-9737-0283eca834e2" containerName="payload-processing" Nov 03 22:51:46.530468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.530445 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.547584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.547303 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r"] Nov 03 22:51:46.551995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.551897 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4f9bb0\"/\"kube-root-ca.crt\"" Nov 03 22:51:46.551995 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.551998 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4f9bb0\"/\"openshift-service-ca.crt\"" Nov 03 22:51:46.559606 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.559587 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4f9bb0\"/\"auth-test-model-4f9bb0-kserve-self-signed-certs\"" Nov 03 22:51:46.559914 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.559884 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4f9bb0\"/\"default-dockercfg-58gvb\"" Nov 03 22:51:46.613279 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.613249 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-dshm\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.613456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.613286 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-home\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.613456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.613392 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tmp-dir\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.613650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.613627 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-model-cache\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.613740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.613708 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzr6\" (UniqueName: \"kubernetes.io/projected/7bee8ce9-e7a9-4119-a3e5-87913b61099d-kube-api-access-tlzr6\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.613823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.613806 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tls-certs\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.715512 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.715474 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-model-cache\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.715706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.715525 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzr6\" (UniqueName: \"kubernetes.io/projected/7bee8ce9-e7a9-4119-a3e5-87913b61099d-kube-api-access-tlzr6\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.715706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.715563 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tls-certs\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.715905 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.715769 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-dshm\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.715905 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.715807 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-home\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.716018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.715906 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tmp-dir\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.716018 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.715939 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-model-cache\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.716189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.716167 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-home\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.716249 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.716177 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tmp-dir\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.718051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.718026 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-dshm\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.718181 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.718167 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tls-certs\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.738497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.738468 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzr6\" (UniqueName: \"kubernetes.io/projected/7bee8ce9-e7a9-4119-a3e5-87913b61099d-kube-api-access-tlzr6\") pod \"auth-test-model-4f9bb0-kserve-f546cf865-bm29r\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:46.861742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:46.857862 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:47.203833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:47.203804 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r"] Nov 03 22:51:47.206041 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:51:47.206005 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bee8ce9_e7a9_4119_a3e5_87913b61099d.slice/crio-780d4148340a4f7c900fae08f093ca6813f0e349e58838123cea66ec7803d164 WatchSource:0}: Error finding container 780d4148340a4f7c900fae08f093ca6813f0e349e58838123cea66ec7803d164: Status 404 returned error can't find the container with id 780d4148340a4f7c900fae08f093ca6813f0e349e58838123cea66ec7803d164 Nov 03 22:51:47.357443 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:47.357405 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" event={"ID":"7bee8ce9-e7a9-4119-a3e5-87913b61099d","Type":"ContainerStarted","Data":"1c33535da131eaedf82d2afcc62ab37eaf97dc88b1705edcb95acbba9e214463"} Nov 03 22:51:47.357624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:47.357457 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" event={"ID":"7bee8ce9-e7a9-4119-a3e5-87913b61099d","Type":"ContainerStarted","Data":"780d4148340a4f7c900fae08f093ca6813f0e349e58838123cea66ec7803d164"} Nov 03 22:51:47.398557 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:47.398501 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" podStartSLOduration=1.398487774 podStartE2EDuration="1.398487774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:47.397439679 +0000 UTC m=+2328.826775668" watchObservedRunningTime="2026-11-03 22:51:47.398487774 +0000 UTC m=+2328.827823759" Nov 03 22:51:47.858562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:47.858534 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:56.858712 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:56.858676 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:56.883588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:56.883553 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:51:56.899950 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:56.899920 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx"] Nov 03 22:51:56.900204 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:56.900180 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" containerID="cri-o://ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf" gracePeriod=30 Nov 03 22:51:57.511176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:51:57.511142 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:52:02.439060 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:52:02.438988 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:52:02.504182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.503840 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:52:02.604288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.601702 4625 generic.go:356] "Generic (PLEG): container finished" podID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerID="ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf" exitCode=0 Nov 03 22:52:02.604288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.601795 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" event={"ID":"0f5bba7c-41a2-4199-9577-fa5eddbf1cde","Type":"ContainerDied","Data":"ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf"} Nov 03 22:52:02.604288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.601847 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" event={"ID":"0f5bba7c-41a2-4199-9577-fa5eddbf1cde","Type":"ContainerDied","Data":"d72b2f45d592d915d7e7c171a99245fada773520a6611f6379cd8dc13a55754d"} Nov 03 22:52:02.604288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.601875 4625 scope.go:122] "RemoveContainer" containerID="ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf" Nov 03 22:52:02.604288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.602033 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx" Nov 03 22:52:02.615209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.615189 4625 scope.go:122] "RemoveContainer" containerID="ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689" Nov 03 22:52:02.625491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.625472 4625 scope.go:122] "RemoveContainer" containerID="ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf" Nov 03 22:52:02.625789 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:52:02.625771 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf\": container with ID starting with ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf not found: ID does not exist" containerID="ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf" Nov 03 22:52:02.625840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.625799 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf"} err="failed to get container status \"ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf\": rpc error: code = NotFound desc = could not find container \"ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf\": container with ID starting with ee47d21d41dcbec29d1f7f1fed7fba9ea20a48971546c7799b51d086063decaf not found: ID does not exist" Nov 03 22:52:02.625840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.625816 4625 scope.go:122] "RemoveContainer" containerID="ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689" Nov 03 22:52:02.626057 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:52:02.626041 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689\": container with ID starting with ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689 not found: ID does not exist" containerID="ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689" Nov 03 22:52:02.626103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.626063 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689"} err="failed to get container status \"ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689\": rpc error: code = NotFound desc = could not find container \"ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689\": container with ID starting with ab99261686cdac8a09056943e1a57309e8135ab482acac2e0837f9f36eabc689 not found: ID does not exist" Nov 03 22:52:02.693247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693158 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-data\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693205 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-podinfo\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693356 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-token\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-token\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693414 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-data" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.693475 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693452 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istiod-ca-cert\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693487 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-envoy\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693518 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-socket\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693637 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693614 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-kube-api-access-sgcg5\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-kube-api-access-sgcg5\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693655 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-credential-socket\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693774 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-certs\") pod \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\" (UID: \"0f5bba7c-41a2-4199-9577-fa5eddbf1cde\") " Nov 03 22:52:02.693941 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693843 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istiod-ca-cert" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:02.694023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.693997 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-credential-socket" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.694099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.694076 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-socket" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.694240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.694219 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-certs" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.694507 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.694487 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.694562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.694513 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.694562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.694526 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.694562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.694539 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.694562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.694550 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.695674 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.695645 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-token" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:02.695856 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.695837 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-podinfo" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:02.695920 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.695882 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-envoy" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.695976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.695963 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-kube-api-access-sgcg5" pod "0f5bba7c-41a2-4199-9577-fa5eddbf1cde" (UID: "0f5bba7c-41a2-4199-9577-fa5eddbf1cde"). InnerVolumeSpecName "kube-api-access-sgcg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:02.812748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.812663 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.812748 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.812696 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.849186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.849152 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.849396 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.849211 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgcg5\" (UniqueName: \"kubernetes.io/projected/0f5bba7c-41a2-4199-9577-fa5eddbf1cde-kube-api-access-sgcg5\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.945343 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.945258 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx"] Nov 03 22:52:02.987369 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:02.979485 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-56bcx"] Nov 03 22:52:04.729782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:04.729711 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" path="/var/lib/kubelet/pods/0f5bba7c-41a2-4199-9577-fa5eddbf1cde/volumes" Nov 03 22:52:13.687138 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:13.687104 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt"] Nov 03 22:52:13.687664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:13.687322 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" podUID="95233eed-0921-4a71-9cf8-3b48f7958b4d" containerName="maas-api" containerID="cri-o://23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307" gracePeriod=30 Nov 03 22:52:14.170001 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.167189 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:52:14.302830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.296797 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls\") pod \"95233eed-0921-4a71-9cf8-3b48f7958b4d\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " Nov 03 22:52:14.302830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.297002 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-metrics-certs\") pod \"95233eed-0921-4a71-9cf8-3b48f7958b4d\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " Nov 03 22:52:14.302830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.297063 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95233eed-0921-4a71-9cf8-3b48f7958b4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95233eed-0921-4a71-9cf8-3b48f7958b4d-openshift-service-ca\") pod \"95233eed-0921-4a71-9cf8-3b48f7958b4d\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " Nov 03 22:52:14.302830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.297226 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95233eed-0921-4a71-9cf8-3b48f7958b4d-kube-api-access-9kn8d\" (UniqueName: \"kubernetes.io/projected/95233eed-0921-4a71-9cf8-3b48f7958b4d-kube-api-access-9kn8d\") pod \"95233eed-0921-4a71-9cf8-3b48f7958b4d\" (UID: \"95233eed-0921-4a71-9cf8-3b48f7958b4d\") " Nov 03 22:52:14.302830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.299754 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95233eed-0921-4a71-9cf8-3b48f7958b4d-openshift-service-ca" pod "95233eed-0921-4a71-9cf8-3b48f7958b4d" (UID: "95233eed-0921-4a71-9cf8-3b48f7958b4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:14.314526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.314409 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-metrics-certs" pod "95233eed-0921-4a71-9cf8-3b48f7958b4d" (UID: "95233eed-0921-4a71-9cf8-3b48f7958b4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:14.315172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.315132 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95233eed-0921-4a71-9cf8-3b48f7958b4d-kube-api-access-9kn8d" pod "95233eed-0921-4a71-9cf8-3b48f7958b4d" (UID: "95233eed-0921-4a71-9cf8-3b48f7958b4d"). InnerVolumeSpecName "kube-api-access-9kn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:14.317198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.317158 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls" pod "95233eed-0921-4a71-9cf8-3b48f7958b4d" (UID: "95233eed-0921-4a71-9cf8-3b48f7958b4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:14.404422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.404381 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:14.404422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.404423 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95233eed-0921-4a71-9cf8-3b48f7958b4d-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:14.404689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.404440 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kn8d\" (UniqueName: \"kubernetes.io/projected/95233eed-0921-4a71-9cf8-3b48f7958b4d-kube-api-access-9kn8d\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:14.404689 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.404454 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95233eed-0921-4a71-9cf8-3b48f7958b4d-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:14.789686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.789651 4625 generic.go:356] "Generic (PLEG): container finished" podID="95233eed-0921-4a71-9cf8-3b48f7958b4d" containerID="23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307" exitCode=0 Nov 03 22:52:14.790176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.789707 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" Nov 03 22:52:14.790176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.789746 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" event={"ID":"95233eed-0921-4a71-9cf8-3b48f7958b4d","Type":"ContainerDied","Data":"23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307"} Nov 03 22:52:14.790176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.789812 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt" event={"ID":"95233eed-0921-4a71-9cf8-3b48f7958b4d","Type":"ContainerDied","Data":"b5ddcf60c31065e4514eb18976efce94c0517f4de1c4aa3924b3e3877e18c192"} Nov 03 22:52:14.790176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.789848 4625 scope.go:122] "RemoveContainer" containerID="23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307" Nov 03 22:52:14.800790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.800768 4625 scope.go:122] "RemoveContainer" containerID="23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307" Nov 03 22:52:14.801077 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:52:14.801058 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307\": container with ID starting with 23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307 not found: ID does not exist" containerID="23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307" Nov 03 22:52:14.801146 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.801089 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307"} err="failed to get container status \"23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307\": rpc error: code = NotFound desc = could not find container \"23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307\": container with ID starting with 23dfa6dc8ef7e52ec1ae28fdab4f98808528abf3d18af5ab218f23959cff2307 not found: ID does not exist" Nov 03 22:52:14.842194 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.842161 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt"] Nov 03 22:52:14.853436 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:14.853408 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b9b5-5b4d587f45-78rnt"] Nov 03 22:52:16.730310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:16.730281 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95233eed-0921-4a71-9cf8-3b48f7958b4d" path="/var/lib/kubelet/pods/95233eed-0921-4a71-9cf8-3b48f7958b4d/volumes" Nov 03 22:52:18.913855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.913821 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8"] Nov 03 22:52:18.914539 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914525 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" Nov 03 22:52:18.914592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914540 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" Nov 03 22:52:18.914592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914553 4625 cpu_manager.go:414] "containerMap: removing container" podUID="95233eed-0921-4a71-9cf8-3b48f7958b4d" containerName="maas-api" Nov 03 22:52:18.914592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914558 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95233eed-0921-4a71-9cf8-3b48f7958b4d" containerName="maas-api" Nov 03 22:52:18.914592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914575 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" Nov 03 22:52:18.914592 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914580 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" Nov 03 22:52:18.914770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914681 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" Nov 03 22:52:18.914770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914690 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="95233eed-0921-4a71-9cf8-3b48f7958b4d" containerName="maas-api" Nov 03 22:52:18.914770 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.914701 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5bba7c-41a2-4199-9577-fa5eddbf1cde" containerName="istio-proxy" Nov 03 22:52:18.919792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.919772 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:18.943317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.943283 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8"] Nov 03 22:52:18.945779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:18.945753 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-9b841d-serving-cert\"" Nov 03 22:52:19.075488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.075453 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-kube-api-access-m7ndb\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.075700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.075511 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.075700 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.075538 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.075885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.075807 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.181970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.181872 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-kube-api-access-m7ndb\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.181970 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.181963 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.182206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.182003 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.182206 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.182122 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.199743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.183418 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.199743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.185924 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.217830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.217760 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-kube-api-access-m7ndb\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.222735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.220345 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.235271 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.234980 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:19.580595 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.580562 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8"] Nov 03 22:52:19.581669 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:52:19.581645 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431b53a2_7fb3_42eb_9e1f_6ac36a1dc51a.slice/crio-32bb7a045b2c754f011557f6eabc55e1b5e216f0b8fb7cff9e1887ec4da7d964 WatchSource:0}: Error finding container 32bb7a045b2c754f011557f6eabc55e1b5e216f0b8fb7cff9e1887ec4da7d964: Status 404 returned error can't find the container with id 32bb7a045b2c754f011557f6eabc55e1b5e216f0b8fb7cff9e1887ec4da7d964 Nov 03 22:52:19.831270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:19.831166 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" event={"ID":"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a","Type":"ContainerStarted","Data":"32bb7a045b2c754f011557f6eabc55e1b5e216f0b8fb7cff9e1887ec4da7d964"} Nov 03 22:52:20.903888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:20.903575 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" event={"ID":"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a","Type":"ContainerStarted","Data":"f64ddf5641f9bf9e15fae8ec98df61a528a86685ef98dc0bcba2399136a9bd11"} Nov 03 22:52:20.903888 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:20.903659 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:20.948753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:20.948667 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" podStartSLOduration=2.615536074 podStartE2EDuration="2.948651172s" totalImagesPullingTime="333.115098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:20.948277614 +0000 UTC m=+2362.377613618" watchObservedRunningTime="2026-11-03 22:52:20.948651172 +0000 UTC m=+2362.377987164" Nov 03 22:52:22.096288 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:22.096247 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv"] Nov 03 22:52:22.096789 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:22.096508 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" containerID="cri-o://9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119" gracePeriod=30 Nov 03 22:52:24.838928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:24.838865 4625 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:24.839342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:24.838964 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:26.917306 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:26.917277 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 22:52:27.380483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.380459 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:52:27.470562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.470474 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-token\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-token\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.470775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.470566 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-data\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.470775 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.470609 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istiod-ca-cert\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.470975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.470777 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-kube-api-access-kvt9t\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-kube-api-access-kvt9t\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.470975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.470850 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-socket\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.470975 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.470907 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-credential-socket\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.471149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.471041 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istiod-ca-cert" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:27.471209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.471181 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-credential-socket" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:27.471351 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.471330 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-socket" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:27.471421 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.471346 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-envoy\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.471640 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.471595 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-certs\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.471753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.471735 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-podinfo\") pod \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\" (UID: \"922cd4bf-ef5a-42a1-9e00-d44e9a707d53\") " Nov 03 22:52:27.472490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.472464 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.472588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.472493 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.472588 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.472507 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.473096 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.473060 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-certs" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:27.473412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.473390 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-data" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:27.473841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.473817 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-envoy" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:27.474823 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.474795 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-podinfo" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:27.477189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.477153 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-kube-api-access-kvt9t" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "kube-api-access-kvt9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:27.477295 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.477203 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-token" pod "922cd4bf-ef5a-42a1-9e00-d44e9a707d53" (UID: "922cd4bf-ef5a-42a1-9e00-d44e9a707d53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:27.574254 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.574221 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.574254 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.574247 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.574254 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.574256 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.574254 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.574264 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.574591 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.574272 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvt9t\" (UniqueName: \"kubernetes.io/projected/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-kube-api-access-kvt9t\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.574591 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.574282 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922cd4bf-ef5a-42a1-9e00-d44e9a707d53-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:27.980023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.979979 4625 generic.go:356] "Generic (PLEG): container finished" podID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerID="9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119" exitCode=0 Nov 03 22:52:27.980467 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.980046 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" Nov 03 22:52:27.980467 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.980064 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" event={"ID":"922cd4bf-ef5a-42a1-9e00-d44e9a707d53","Type":"ContainerDied","Data":"9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119"} Nov 03 22:52:27.980467 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.980117 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv" event={"ID":"922cd4bf-ef5a-42a1-9e00-d44e9a707d53","Type":"ContainerDied","Data":"1ca2961589583f0ac38804c78dbe8d93c7cb16214c7a0d0bed4cad8e6c932861"} Nov 03 22:52:27.980467 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.980157 4625 scope.go:122] "RemoveContainer" containerID="9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119" Nov 03 22:52:27.992998 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:27.992974 4625 scope.go:122] "RemoveContainer" containerID="356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead" Nov 03 22:52:28.002372 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:28.002351 4625 scope.go:122] "RemoveContainer" containerID="9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119" Nov 03 22:52:28.002655 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:52:28.002636 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119\": container with ID starting with 9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119 not found: ID does not exist" containerID="9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119" Nov 03 22:52:28.002728 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:28.002664 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119"} err="failed to get container status \"9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119\": rpc error: code = NotFound desc = could not find container \"9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119\": container with ID starting with 9c927702d58cde11fb8273f29d0ccf0d2a34753252f772b97573b7d5e2298119 not found: ID does not exist" Nov 03 22:52:28.002728 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:28.002679 4625 scope.go:122] "RemoveContainer" containerID="356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead" Nov 03 22:52:28.002944 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:52:28.002929 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead\": container with ID starting with 356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead not found: ID does not exist" containerID="356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead" Nov 03 22:52:28.002986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:28.002950 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead"} err="failed to get container status \"356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead\": rpc error: code = NotFound desc = could not find container \"356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead\": container with ID starting with 356bfb9e326132eb71d251c52ef00736acade36a7611679d5ecc617e5834eead not found: ID does not exist" Nov 03 22:52:28.037761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:28.037736 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv"] Nov 03 22:52:28.051687 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:28.051658 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-br8fv"] Nov 03 22:52:28.731092 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:28.731056 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" path="/var/lib/kubelet/pods/922cd4bf-ef5a-42a1-9e00-d44e9a707d53/volumes" Nov 03 22:52:32.185893 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:52:32.185856 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:52:36.666570 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:36.666537 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk"] Nov 03 22:52:36.667247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:36.667213 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" podUID="8cca9d4d-9923-47ac-8c95-96ac0376b42f" containerName="maas-api" containerID="cri-o://beebd4d19186ae0563f7c293b5c68a502f206ffbf1376785e3059cb49042858b" gracePeriod=30 Nov 03 22:52:37.228213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.228176 4625 generic.go:356] "Generic (PLEG): container finished" podID="8cca9d4d-9923-47ac-8c95-96ac0376b42f" containerID="beebd4d19186ae0563f7c293b5c68a502f206ffbf1376785e3059cb49042858b" exitCode=0 Nov 03 22:52:37.228392 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.228349 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" event={"ID":"8cca9d4d-9923-47ac-8c95-96ac0376b42f","Type":"ContainerDied","Data":"beebd4d19186ae0563f7c293b5c68a502f206ffbf1376785e3059cb49042858b"} Nov 03 22:52:37.248456 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.248418 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:52:37.339983 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.339950 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cca9d4d-9923-47ac-8c95-96ac0376b42f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cca9d4d-9923-47ac-8c95-96ac0376b42f-openshift-service-ca\") pod \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " Nov 03 22:52:37.340377 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.340343 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cca9d4d-9923-47ac-8c95-96ac0376b42f-kube-api-access-7589j\" (UniqueName: \"kubernetes.io/projected/8cca9d4d-9923-47ac-8c95-96ac0376b42f-kube-api-access-7589j\") pod \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " Nov 03 22:52:37.340589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.340570 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-metrics-certs\") pod \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " Nov 03 22:52:37.340787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.340762 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-maas-api-tls\") pod \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\" (UID: \"8cca9d4d-9923-47ac-8c95-96ac0376b42f\") " Nov 03 22:52:37.340926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.340380 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cca9d4d-9923-47ac-8c95-96ac0376b42f-openshift-service-ca" pod "8cca9d4d-9923-47ac-8c95-96ac0376b42f" (UID: "8cca9d4d-9923-47ac-8c95-96ac0376b42f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:37.341494 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.341471 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cca9d4d-9923-47ac-8c95-96ac0376b42f-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:37.349149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.349123 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-metrics-certs" pod "8cca9d4d-9923-47ac-8c95-96ac0376b42f" (UID: "8cca9d4d-9923-47ac-8c95-96ac0376b42f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:37.349310 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.349288 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cca9d4d-9923-47ac-8c95-96ac0376b42f-kube-api-access-7589j" pod "8cca9d4d-9923-47ac-8c95-96ac0376b42f" (UID: "8cca9d4d-9923-47ac-8c95-96ac0376b42f"). InnerVolumeSpecName "kube-api-access-7589j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:37.350833 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.350795 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-maas-api-tls" pod "8cca9d4d-9923-47ac-8c95-96ac0376b42f" (UID: "8cca9d4d-9923-47ac-8c95-96ac0376b42f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:37.443119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.443082 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:37.443119 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.443117 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7589j\" (UniqueName: \"kubernetes.io/projected/8cca9d4d-9923-47ac-8c95-96ac0376b42f-kube-api-access-7589j\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:37.443319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:37.443134 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cca9d4d-9923-47ac-8c95-96ac0376b42f-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:52:38.237168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:38.237116 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" Nov 03 22:52:38.237665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:38.237122 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk" event={"ID":"8cca9d4d-9923-47ac-8c95-96ac0376b42f","Type":"ContainerDied","Data":"39cb02f9f334421339cc2911f46a1d8c5e313f5e20c20b841c006058567d9b67"} Nov 03 22:52:38.237665 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:38.237276 4625 scope.go:122] "RemoveContainer" containerID="beebd4d19186ae0563f7c293b5c68a502f206ffbf1376785e3059cb49042858b" Nov 03 22:52:38.289249 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:38.289207 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk"] Nov 03 22:52:38.300754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:38.300711 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccdb60-5cfb4d5589-mdchk"] Nov 03 22:52:38.730502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:52:38.730463 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cca9d4d-9923-47ac-8c95-96ac0376b42f" path="/var/lib/kubelet/pods/8cca9d4d-9923-47ac-8c95-96ac0376b42f/volumes" Nov 03 22:53:02.190625 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:53:02.190588 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:53:02.505985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:02.505886 4625 generic.go:356] "Generic (PLEG): container finished" podID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerID="387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209" exitCode=0 Nov 03 22:53:02.505985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:02.505954 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" event={"ID":"3f93335b-e263-455b-a2eb-d34acc1f1398","Type":"ContainerDied","Data":"387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209"} Nov 03 22:53:02.506422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:02.506407 4625 scope.go:122] "RemoveContainer" containerID="387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209" Nov 03 22:53:02.508292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:02.508262 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:53:02.508371 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:02.508333 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:53:02.508415 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:02.508372 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:53:03.516233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:03.516201 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" event={"ID":"3f93335b-e263-455b-a2eb-d34acc1f1398","Type":"ContainerStarted","Data":"b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a"} Nov 03 22:53:03.808482 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:03.808394 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:53:03.859740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:03.856632 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:53:04.170761 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.170675 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22"] Nov 03 22:53:04.171287 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.171230 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" podUID="eaf35d5f-1526-42ce-b8af-1f400197dc3d" containerName="maas-api" containerID="cri-o://efe053106a8d99151a2aad1f1434486c3020eef297a128d91ccce52f1f5b4e72" gracePeriod=30 Nov 03 22:53:04.223638 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.223560 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:53:04.224389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.224357 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:53:04.224502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.224424 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 22:53:04.905548 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.905521 4625 generic.go:356] "Generic (PLEG): container finished" podID="eaf35d5f-1526-42ce-b8af-1f400197dc3d" containerID="efe053106a8d99151a2aad1f1434486c3020eef297a128d91ccce52f1f5b4e72" exitCode=0 Nov 03 22:53:04.906940 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.906913 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" event={"ID":"eaf35d5f-1526-42ce-b8af-1f400197dc3d","Type":"ContainerDied","Data":"efe053106a8d99151a2aad1f1434486c3020eef297a128d91ccce52f1f5b4e72"} Nov 03 22:53:04.997242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:04.997210 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:53:05.117626 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.117582 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-metrics-certs\") pod \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " Nov 03 22:53:05.117845 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.117758 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf35d5f-1526-42ce-b8af-1f400197dc3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf35d5f-1526-42ce-b8af-1f400197dc3d-openshift-service-ca\") pod \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " Nov 03 22:53:05.119090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.117974 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls\") pod \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " Nov 03 22:53:05.119090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.118140 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf35d5f-1526-42ce-b8af-1f400197dc3d-openshift-service-ca" pod "eaf35d5f-1526-42ce-b8af-1f400197dc3d" (UID: "eaf35d5f-1526-42ce-b8af-1f400197dc3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:05.119090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.118370 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf35d5f-1526-42ce-b8af-1f400197dc3d-kube-api-access-2p94k\" (UniqueName: \"kubernetes.io/projected/eaf35d5f-1526-42ce-b8af-1f400197dc3d-kube-api-access-2p94k\") pod \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\" (UID: \"eaf35d5f-1526-42ce-b8af-1f400197dc3d\") " Nov 03 22:53:05.119090 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.119035 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf35d5f-1526-42ce-b8af-1f400197dc3d-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:05.120224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.120190 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls" pod "eaf35d5f-1526-42ce-b8af-1f400197dc3d" (UID: "eaf35d5f-1526-42ce-b8af-1f400197dc3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:05.120329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.120275 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-metrics-certs" pod "eaf35d5f-1526-42ce-b8af-1f400197dc3d" (UID: "eaf35d5f-1526-42ce-b8af-1f400197dc3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:05.120509 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.120492 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf35d5f-1526-42ce-b8af-1f400197dc3d-kube-api-access-2p94k" pod "eaf35d5f-1526-42ce-b8af-1f400197dc3d" (UID: "eaf35d5f-1526-42ce-b8af-1f400197dc3d"). InnerVolumeSpecName "kube-api-access-2p94k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:05.219976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.219894 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:05.219976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.219925 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p94k\" (UniqueName: \"kubernetes.io/projected/eaf35d5f-1526-42ce-b8af-1f400197dc3d-kube-api-access-2p94k\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:05.219976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.219935 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf35d5f-1526-42ce-b8af-1f400197dc3d-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:05.223180 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.223153 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:53:05.223299 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.223226 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 22:53:05.915762 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.915690 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" event={"ID":"eaf35d5f-1526-42ce-b8af-1f400197dc3d","Type":"ContainerDied","Data":"38faf7f7d3dec3bc2694d5c041f5341b2df1a80bc6498a7ea61ce0afd5634705"} Nov 03 22:53:05.916176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.915765 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22" Nov 03 22:53:05.916176 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.915797 4625 scope.go:122] "RemoveContainer" containerID="efe053106a8d99151a2aad1f1434486c3020eef297a128d91ccce52f1f5b4e72" Nov 03 22:53:05.971157 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.971046 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22"] Nov 03 22:53:05.982235 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:05.982208 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe7ca0-6b69659b9-75m22"] Nov 03 22:53:06.214565 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:06.214490 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:53:06.223682 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:06.223658 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:53:06.223812 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:06.223730 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 22:53:06.730450 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:06.730418 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf35d5f-1526-42ce-b8af-1f400197dc3d" path="/var/lib/kubelet/pods/eaf35d5f-1526-42ce-b8af-1f400197dc3d/volumes" Nov 03 22:53:07.224020 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:07.223986 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:53:07.224415 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:07.224035 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 22:53:08.223706 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.223672 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:53:08.223894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.223772 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 22:53:08.332951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.332888 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh"] Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.333947 4625 cpu_manager.go:414] "containerMap: removing container" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.333974 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.334016 4625 cpu_manager.go:414] "containerMap: removing container" podUID="eaf35d5f-1526-42ce-b8af-1f400197dc3d" containerName="maas-api" Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.334024 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf35d5f-1526-42ce-b8af-1f400197dc3d" containerName="maas-api" Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.334045 4625 cpu_manager.go:414] "containerMap: removing container" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.334052 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.334105 4625 cpu_manager.go:414] "containerMap: removing container" podUID="8cca9d4d-9923-47ac-8c95-96ac0376b42f" containerName="maas-api" Nov 03 22:53:08.336754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.334113 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cca9d4d-9923-47ac-8c95-96ac0376b42f" containerName="maas-api" Nov 03 22:53:08.351795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.347938 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" Nov 03 22:53:08.351795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.347983 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cca9d4d-9923-47ac-8c95-96ac0376b42f" containerName="maas-api" Nov 03 22:53:08.351795 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.348014 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf35d5f-1526-42ce-b8af-1f400197dc3d" containerName="maas-api" Nov 03 22:53:08.386787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.385281 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.386787 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.386221 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh"] Nov 03 22:53:08.416346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.414700 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"default-dockercfg-xckd5\"" Nov 03 22:53:08.416346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.414858 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"auth-test-model-23d281-kserve-self-signed-certs\"" Nov 03 22:53:08.429121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.428853 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"openshift-service-ca.crt\"" Nov 03 22:53:08.429121 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.428797 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"kube-root-ca.crt\"" Nov 03 22:53:08.489839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.486357 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-dshm\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.489839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.486706 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tmp-dir\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.489839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.486792 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-model-cache\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.489839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.486871 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-home\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.489839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.486922 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tls-certs\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.489839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.486992 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwxhb\" (UniqueName: \"kubernetes.io/projected/9f244f35-64d5-4c80-9f9b-41c558aae6a9-kube-api-access-rwxhb\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.595763 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tmp-dir\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.595865 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-model-cache\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.595941 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-home\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.595988 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tls-certs\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.596054 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwxhb\" (UniqueName: \"kubernetes.io/projected/9f244f35-64d5-4c80-9f9b-41c558aae6a9-kube-api-access-rwxhb\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.596116 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-dshm\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.596296 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tmp-dir\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.596566 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-home\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.608380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.596820 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-model-cache\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.660779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.635196 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tls-certs\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.660779 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.639154 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwxhb\" (UniqueName: \"kubernetes.io/projected/9f244f35-64d5-4c80-9f9b-41c558aae6a9-kube-api-access-rwxhb\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.667479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.667447 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-dshm\") pod \"auth-test-model-23d281-kserve-78dcbd5765-f2fxh\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:08.730389 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:08.730363 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:09.055497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:09.053480 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh"] Nov 03 22:53:09.062316 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:53:09.062287 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f244f35_64d5_4c80_9f9b_41c558aae6a9.slice/crio-425653f384798c0059d2026d997ea8812d49c48de3418a79953101e6b768e443 WatchSource:0}: Error finding container 425653f384798c0059d2026d997ea8812d49c48de3418a79953101e6b768e443: Status 404 returned error can't find the container with id 425653f384798c0059d2026d997ea8812d49c48de3418a79953101e6b768e443 Nov 03 22:53:09.228623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:09.228589 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:53:09.229691 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:09.229666 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 22:53:09.974231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:09.974191 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" event={"ID":"9f244f35-64d5-4c80-9f9b-41c558aae6a9","Type":"ContainerStarted","Data":"a1f5b2b71d61ece52822d9f3f1c16ebf03b23504741dff8bd30fc35dda193ab4"} Nov 03 22:53:09.974632 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:09.974242 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" event={"ID":"9f244f35-64d5-4c80-9f9b-41c558aae6a9","Type":"ContainerStarted","Data":"425653f384798c0059d2026d997ea8812d49c48de3418a79953101e6b768e443"} Nov 03 22:53:10.015688 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:10.015634 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" podStartSLOduration=2.015621289 podStartE2EDuration="2.015621289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:10.014007094 +0000 UTC m=+2411.443343082" watchObservedRunningTime="2026-11-03 22:53:10.015621289 +0000 UTC m=+2411.444957280" Nov 03 22:53:10.731266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:10.731236 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:11.925605 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:11.925546 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs"] Nov 03 22:53:11.926103 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:11.925793 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" containerID="cri-o://e5d3792e36390b60d3409c67ca4a394ad2f60f55a7136cf7bacdba903524f0ac" gracePeriod=30 Nov 03 22:53:17.132302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.132226 4625 generic.go:356] "Generic (PLEG): container finished" podID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerID="e5d3792e36390b60d3409c67ca4a394ad2f60f55a7136cf7bacdba903524f0ac" exitCode=0 Nov 03 22:53:17.133040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.132465 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" event={"ID":"9a9758dc-7a16-4cab-bca6-3a3833037426","Type":"ContainerDied","Data":"e5d3792e36390b60d3409c67ca4a394ad2f60f55a7136cf7bacdba903524f0ac"} Nov 03 22:53:17.133234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.133220 4625 scope.go:122] "RemoveContainer" containerID="cdcfa9957f998573419b9464f3595afd1139e32ca8a82159e9a59ff3c301f2e1" Nov 03 22:53:17.209459 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.209435 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269166 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-envoy\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269220 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9758dc-7a16-4cab-bca6-3a3833037426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9758dc-7a16-4cab-bca6-3a3833037426-istiod-ca-cert\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269256 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-token\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-token\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269408 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-socket\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269525 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-podinfo\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269572 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-kube-api-access-qqxgf\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-kube-api-access-qqxgf\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269702 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-credential-socket\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269832 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-data\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.269984 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-certs\") pod \"9a9758dc-7a16-4cab-bca6-3a3833037426\" (UID: \"9a9758dc-7a16-4cab-bca6-3a3833037426\") " Nov 03 22:53:17.270993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.270471 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-data" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:17.270993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.270581 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-credential-socket" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:17.270993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.270648 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-socket" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:17.270993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.270891 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-certs" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:17.271195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.271041 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9758dc-7a16-4cab-bca6-3a3833037426-istiod-ca-cert" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:17.271874 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.271840 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.272011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.271877 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.272011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.271891 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.272011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.271903 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9758dc-7a16-4cab-bca6-3a3833037426-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.272011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.271915 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.273933 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.273899 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-token" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:17.276976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.275575 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-kube-api-access-qqxgf" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "kube-api-access-qqxgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:17.276976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.275687 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-podinfo" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:17.276976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.275968 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-envoy" pod "9a9758dc-7a16-4cab-bca6-3a3833037426" (UID: "9a9758dc-7a16-4cab-bca6-3a3833037426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:17.373342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.373304 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.373342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.373333 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.373342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.373344 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9758dc-7a16-4cab-bca6-3a3833037426-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:17.373582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:17.373357 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqxgf\" (UniqueName: \"kubernetes.io/projected/9a9758dc-7a16-4cab-bca6-3a3833037426-kube-api-access-qqxgf\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:53:18.181842 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.178270 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" event={"ID":"9a9758dc-7a16-4cab-bca6-3a3833037426","Type":"ContainerDied","Data":"7a66ac045b6d71b422b421f559d1892042a9770acfaf047a4afcaf371b9e7928"} Nov 03 22:53:18.181842 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.178341 4625 scope.go:122] "RemoveContainer" containerID="e5d3792e36390b60d3409c67ca4a394ad2f60f55a7136cf7bacdba903524f0ac" Nov 03 22:53:18.181842 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.178460 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs" Nov 03 22:53:18.245085 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.245057 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs"] Nov 03 22:53:18.271458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.271415 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fe7ca0-openshift-default-7598bdb7d5-jqmrs"] Nov 03 22:53:18.731025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.730993 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" path="/var/lib/kubelet/pods/9a9758dc-7a16-4cab-bca6-3a3833037426/volumes" Nov 03 22:53:18.731405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.731387 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:18.748911 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:18.748885 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:19.205896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:19.205864 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:53:32.187265 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:53:32.187223 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:53:43.624559 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.624526 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w"] Nov 03 22:53:43.625259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.625241 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" Nov 03 22:53:43.625313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.625260 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" Nov 03 22:53:43.625313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.625270 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" Nov 03 22:53:43.625313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.625277 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" Nov 03 22:53:43.625413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.625386 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" Nov 03 22:53:43.625413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.625393 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9758dc-7a16-4cab-bca6-3a3833037426" containerName="istio-proxy" Nov 03 22:53:43.625413 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.625400 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="922cd4bf-ef5a-42a1-9e00-d44e9a707d53" containerName="istio-proxy" Nov 03 22:53:43.628447 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.628428 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.657957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.657910 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"kube-root-ca.crt\"" Nov 03 22:53:43.657957 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.657935 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"openshift-service-ca.crt\"" Nov 03 22:53:43.658290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.658268 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"sub-test-model-2a2eca-kserve-self-signed-certs\"" Nov 03 22:53:43.658290 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.658285 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"default-dockercfg-6fnmp\"" Nov 03 22:53:43.683479 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.683435 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w"] Nov 03 22:53:43.697234 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.697194 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-model-cache\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.697426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.697274 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5692af-1f8c-41db-bc03-7c53104de383-tls-certs\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.697426 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.697383 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl79q\" (UniqueName: \"kubernetes.io/projected/6f5692af-1f8c-41db-bc03-7c53104de383-kube-api-access-tl79q\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.697543 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.697454 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-tmp-dir\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.697667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.697550 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-home\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.697753 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.697668 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-dshm\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.799627 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.799587 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-model-cache\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.799840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.799642 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5692af-1f8c-41db-bc03-7c53104de383-tls-certs\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.799840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.799684 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl79q\" (UniqueName: \"kubernetes.io/projected/6f5692af-1f8c-41db-bc03-7c53104de383-kube-api-access-tl79q\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.799840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.799733 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-tmp-dir\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.799840 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.799822 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-home\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.800104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.799947 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-dshm\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.800104 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.800091 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-model-cache\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.800226 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.800162 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-home\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.800345 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.800317 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-tmp-dir\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.802053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.802032 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-dshm\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.802313 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.802297 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5692af-1f8c-41db-bc03-7c53104de383-tls-certs\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.840832 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.840799 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl79q\" (UniqueName: \"kubernetes.io/projected/6f5692af-1f8c-41db-bc03-7c53104de383-kube-api-access-tl79q\") pod \"sub-test-model-2a2eca-kserve-788cf5788f-6nx2w\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:43.951952 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:43.951025 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:44.343807 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:53:44.343772 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f5692af_1f8c_41db_bc03_7c53104de383.slice/crio-97c712d9f07adbb478ea866389cacc84097c8136425ebfa671db5a24bbbf4e99 WatchSource:0}: Error finding container 97c712d9f07adbb478ea866389cacc84097c8136425ebfa671db5a24bbbf4e99: Status 404 returned error can't find the container with id 97c712d9f07adbb478ea866389cacc84097c8136425ebfa671db5a24bbbf4e99 Nov 03 22:53:44.344335 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:44.344308 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w"] Nov 03 22:53:44.498577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:44.498544 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" event={"ID":"6f5692af-1f8c-41db-bc03-7c53104de383","Type":"ContainerStarted","Data":"8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7"} Nov 03 22:53:44.498774 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:44.498586 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" event={"ID":"6f5692af-1f8c-41db-bc03-7c53104de383","Type":"ContainerStarted","Data":"97c712d9f07adbb478ea866389cacc84097c8136425ebfa671db5a24bbbf4e99"} Nov 03 22:53:44.542752 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:44.542674 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" podStartSLOduration=1.5426557239999998 podStartE2EDuration="1.542655724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:44.539903063 +0000 UTC m=+2445.969239055" watchObservedRunningTime="2026-11-03 22:53:44.542655724 +0000 UTC m=+2445.971991717" Nov 03 22:53:44.955944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:44.955915 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:53.956562 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:53.956527 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:53.968972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:53.968948 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:53:54.596280 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:53:54.596240 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:54:02.187119 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:54:02.187080 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:54:04.286002 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.285963 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg"] Nov 03 22:54:04.286617 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.286221 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" podUID="6b316235-3895-40b3-b3dd-623281a7e2dc" containerName="maas-api" containerID="cri-o://e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539" gracePeriod=30 Nov 03 22:54:04.873574 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.873537 4625 generic.go:356] "Generic (PLEG): container finished" podID="6b316235-3895-40b3-b3dd-623281a7e2dc" containerID="e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539" exitCode=0 Nov 03 22:54:04.877193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.875491 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:54:04.886196 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.886161 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" event={"ID":"6b316235-3895-40b3-b3dd-623281a7e2dc","Type":"ContainerDied","Data":"e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539"} Nov 03 22:54:04.886422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.886406 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" event={"ID":"6b316235-3895-40b3-b3dd-623281a7e2dc","Type":"ContainerDied","Data":"0f7e448ebde1ee14d897a6ca9735ebeb800549b031e196eff6c896b4ae297b73"} Nov 03 22:54:04.886537 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.886525 4625 scope.go:122] "RemoveContainer" containerID="e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539" Nov 03 22:54:04.901265 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.901244 4625 scope.go:122] "RemoveContainer" containerID="e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539" Nov 03 22:54:04.901548 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:54:04.901526 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539\": container with ID starting with e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539 not found: ID does not exist" containerID="e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539" Nov 03 22:54:04.901630 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.901556 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539"} err="failed to get container status \"e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539\": rpc error: code = NotFound desc = could not find container \"e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539\": container with ID starting with e97c2f9f01d00e8ec88f5b2f379a6fd786d737f157e25f760a7f7eaaa4e23539 not found: ID does not exist" Nov 03 22:54:04.979899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.976258 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b316235-3895-40b3-b3dd-623281a7e2dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b316235-3895-40b3-b3dd-623281a7e2dc-openshift-service-ca\") pod \"6b316235-3895-40b3-b3dd-623281a7e2dc\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " Nov 03 22:54:04.979899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.976450 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-metrics-certs\") pod \"6b316235-3895-40b3-b3dd-623281a7e2dc\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " Nov 03 22:54:04.979899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.976514 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b316235-3895-40b3-b3dd-623281a7e2dc-kube-api-access-l27xv\" (UniqueName: \"kubernetes.io/projected/6b316235-3895-40b3-b3dd-623281a7e2dc-kube-api-access-l27xv\") pod \"6b316235-3895-40b3-b3dd-623281a7e2dc\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " Nov 03 22:54:04.979899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.976607 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls\") pod \"6b316235-3895-40b3-b3dd-623281a7e2dc\" (UID: \"6b316235-3895-40b3-b3dd-623281a7e2dc\") " Nov 03 22:54:04.979899 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.979589 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b316235-3895-40b3-b3dd-623281a7e2dc-openshift-service-ca" pod "6b316235-3895-40b3-b3dd-623281a7e2dc" (UID: "6b316235-3895-40b3-b3dd-623281a7e2dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:04.983239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.983154 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls" pod "6b316235-3895-40b3-b3dd-623281a7e2dc" (UID: "6b316235-3895-40b3-b3dd-623281a7e2dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:04.985099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.985035 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-metrics-certs" pod "6b316235-3895-40b3-b3dd-623281a7e2dc" (UID: "6b316235-3895-40b3-b3dd-623281a7e2dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:04.985099 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:04.985081 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b316235-3895-40b3-b3dd-623281a7e2dc-kube-api-access-l27xv" pod "6b316235-3895-40b3-b3dd-623281a7e2dc" (UID: "6b316235-3895-40b3-b3dd-623281a7e2dc"). InnerVolumeSpecName "kube-api-access-l27xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:05.083736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:05.083137 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:05.083736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:05.083177 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l27xv\" (UniqueName: \"kubernetes.io/projected/6b316235-3895-40b3-b3dd-623281a7e2dc-kube-api-access-l27xv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:05.083736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:05.083191 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b316235-3895-40b3-b3dd-623281a7e2dc-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:05.083736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:05.083204 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b316235-3895-40b3-b3dd-623281a7e2dc-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:05.884505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:05.884476 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg" Nov 03 22:54:05.951609 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:05.951572 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg"] Nov 03 22:54:05.961111 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:05.961083 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-341c51-7cf9d5c67c-w5hqg"] Nov 03 22:54:06.730437 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:06.730404 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b316235-3895-40b3-b3dd-623281a7e2dc" path="/var/lib/kubelet/pods/6b316235-3895-40b3-b3dd-623281a7e2dc/volumes" Nov 03 22:54:12.607221 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:12.607142 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg"] Nov 03 22:54:12.607640 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:12.607431 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" containerID="cri-o://24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e" gracePeriod=30 Nov 03 22:54:17.895790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.895763 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:54:17.985045 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.985011 4625 generic.go:356] "Generic (PLEG): container finished" podID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerID="24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e" exitCode=0 Nov 03 22:54:17.985317 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.985097 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" Nov 03 22:54:17.985439 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.985098 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" event={"ID":"22fede61-3e56-4ddf-baa0-cfa2359c4556","Type":"ContainerDied","Data":"24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e"} Nov 03 22:54:17.985513 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.985470 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg" event={"ID":"22fede61-3e56-4ddf-baa0-cfa2359c4556","Type":"ContainerDied","Data":"abb612519734e91e29aeb3dcba1703d7c5989e77c6f8844a0ca1b10adf8dc5bf"} Nov 03 22:54:17.985513 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.985502 4625 scope.go:122] "RemoveContainer" containerID="24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e" Nov 03 22:54:17.989273 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989236 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-certs\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.989384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989290 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-credential-socket\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.989384 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989334 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-kube-api-access-pkdvt\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-kube-api-access-pkdvt\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.989521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989466 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-token\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-token\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.989521 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989500 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-socket\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.989657 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989637 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fede61-3e56-4ddf-baa0-cfa2359c4556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fede61-3e56-4ddf-baa0-cfa2359c4556-istiod-ca-cert\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.990686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989697 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-data\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.990686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989775 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-podinfo\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.990686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.989886 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-envoy\") pod \"22fede61-3e56-4ddf-baa0-cfa2359c4556\" (UID: \"22fede61-3e56-4ddf-baa0-cfa2359c4556\") " Nov 03 22:54:17.990686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.990477 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-certs" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:17.991047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.990696 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-credential-socket" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:17.991047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.990888 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-data" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:17.991047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.990903 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-socket" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:17.991519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.991497 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fede61-3e56-4ddf-baa0-cfa2359c4556-istiod-ca-cert" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:17.996916 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.996891 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-token" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:17.997032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.996927 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-kube-api-access-pkdvt" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "kube-api-access-pkdvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:17.997032 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.996996 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-podinfo" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:17.999231 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:17.999206 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-envoy" pod "22fede61-3e56-4ddf-baa0-cfa2359c4556" (UID: "22fede61-3e56-4ddf-baa0-cfa2359c4556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:18.000649 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.000627 4625 scope.go:122] "RemoveContainer" containerID="e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558" Nov 03 22:54:18.027316 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.027286 4625 scope.go:122] "RemoveContainer" containerID="24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e" Nov 03 22:54:18.027639 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:54:18.027617 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e\": container with ID starting with 24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e not found: ID does not exist" containerID="24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e" Nov 03 22:54:18.027711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.027649 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e"} err="failed to get container status \"24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e\": rpc error: code = NotFound desc = could not find container \"24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e\": container with ID starting with 24bc873661a3de2d71b4aec7acf355cf65a3582a7f6a90fa2df9abb9ff8ba02e not found: ID does not exist" Nov 03 22:54:18.027711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.027668 4625 scope.go:122] "RemoveContainer" containerID="e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558" Nov 03 22:54:18.027969 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:54:18.027950 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558\": container with ID starting with e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558 not found: ID does not exist" containerID="e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558" Nov 03 22:54:18.028014 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.027975 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558"} err="failed to get container status \"e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558\": rpc error: code = NotFound desc = could not find container \"e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558\": container with ID starting with e67c8a66315022ab05a29f1bc40dc4066f34864efdc967b66567cba095531558 not found: ID does not exist" Nov 03 22:54:18.091686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091654 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091681 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091690 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fede61-3e56-4ddf-baa0-cfa2359c4556-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091698 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091706 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091750 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091764 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091772 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fede61-3e56-4ddf-baa0-cfa2359c4556-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.091976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.091781 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkdvt\" (UniqueName: \"kubernetes.io/projected/22fede61-3e56-4ddf-baa0-cfa2359c4556-kube-api-access-pkdvt\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:18.344422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.344393 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg"] Nov 03 22:54:18.348432 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.348403 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-341c51-openshift-default-56f6687b6f-dqgpg"] Nov 03 22:54:18.730026 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:18.729987 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" path="/var/lib/kubelet/pods/22fede61-3e56-4ddf-baa0-cfa2359c4556/volumes" Nov 03 22:54:31.284360 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.284319 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9"] Nov 03 22:54:31.285391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285372 4625 cpu_manager.go:414] "containerMap: removing container" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" Nov 03 22:54:31.285391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285392 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" Nov 03 22:54:31.285528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285411 4625 cpu_manager.go:414] "containerMap: removing container" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" Nov 03 22:54:31.285528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285416 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" Nov 03 22:54:31.285528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285437 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6b316235-3895-40b3-b3dd-623281a7e2dc" containerName="maas-api" Nov 03 22:54:31.285528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285442 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b316235-3895-40b3-b3dd-623281a7e2dc" containerName="maas-api" Nov 03 22:54:31.285767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285539 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" Nov 03 22:54:31.285767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285548 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fede61-3e56-4ddf-baa0-cfa2359c4556" containerName="istio-proxy" Nov 03 22:54:31.285767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.285558 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b316235-3895-40b3-b3dd-623281a7e2dc" containerName="maas-api" Nov 03 22:54:31.304735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.291813 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.315400 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.315372 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-9b841d\"/\"kube-root-ca.crt\"" Nov 03 22:54:31.315636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.315617 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-9b841d\"/\"openshift-service-ca.crt\"" Nov 03 22:54:31.317193 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.317168 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9"] Nov 03 22:54:31.318920 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.318899 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-9b841d\"/\"default-dockercfg-ll58d\"" Nov 03 22:54:31.319060 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.319021 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-9b841d\"/\"auth-test-model-9b841d-kserve-self-signed-certs\"" Nov 03 22:54:31.409669 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.409634 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-model-cache\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.409892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.409681 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-tmp-dir\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.409892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.409819 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-home\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.409892 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.409864 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baab7fc-28dd-4df9-a6e1-060b85470d72-tls-certs\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.410010 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.409987 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-dshm\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.410123 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.410100 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479f9\" (UniqueName: \"kubernetes.io/projected/2baab7fc-28dd-4df9-a6e1-060b85470d72-kube-api-access-479f9\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.518738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.515636 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479f9\" (UniqueName: \"kubernetes.io/projected/2baab7fc-28dd-4df9-a6e1-060b85470d72-kube-api-access-479f9\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.518738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.515814 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-model-cache\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.518738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.515857 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-tmp-dir\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.518738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.515907 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-home\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.518738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.515965 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baab7fc-28dd-4df9-a6e1-060b85470d72-tls-certs\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.518738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.516080 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-dshm\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.525747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.525015 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-tmp-dir\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.525747 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.525664 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-model-cache\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.525986 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.525925 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-home\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.569035 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.568380 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baab7fc-28dd-4df9-a6e1-060b85470d72-tls-certs\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.580246 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.580215 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-dshm\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.601872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.601798 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479f9\" (UniqueName: \"kubernetes.io/projected/2baab7fc-28dd-4df9-a6e1-060b85470d72-kube-api-access-479f9\") pod \"auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.630497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.630218 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:31.949102 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:54:31.949073 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2baab7fc_28dd_4df9_a6e1_060b85470d72.slice/crio-9bbbfb4e193ed00fcfa5ca37d672e97d70c6b5be3fa072396b4866e5f9705943 WatchSource:0}: Error finding container 9bbbfb4e193ed00fcfa5ca37d672e97d70c6b5be3fa072396b4866e5f9705943: Status 404 returned error can't find the container with id 9bbbfb4e193ed00fcfa5ca37d672e97d70c6b5be3fa072396b4866e5f9705943 Nov 03 22:54:31.950043 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:31.950012 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9"] Nov 03 22:54:32.132948 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:32.132911 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" event={"ID":"2baab7fc-28dd-4df9-a6e1-060b85470d72","Type":"ContainerStarted","Data":"a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7"} Nov 03 22:54:32.133140 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:32.132961 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" event={"ID":"2baab7fc-28dd-4df9-a6e1-060b85470d72","Type":"ContainerStarted","Data":"9bbbfb4e193ed00fcfa5ca37d672e97d70c6b5be3fa072396b4866e5f9705943"} Nov 03 22:54:32.173792 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:32.173698 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" podStartSLOduration=1.173679884 podStartE2EDuration="1.173679884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:32.17284215 +0000 UTC m=+2493.602178142" watchObservedRunningTime="2026-11-03 22:54:32.173679884 +0000 UTC m=+2493.603015877" Nov 03 22:54:32.189076 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:54:32.189039 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:54:32.632440 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:32.632406 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:34.739623 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:34.739592 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc"] Nov 03 22:54:34.740144 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:34.739852 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" containerID="cri-o://686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212" gracePeriod=30 Nov 03 22:54:35.877056 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:35.877015 4625 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:54:35.877545 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:35.877083 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:54:36.984468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:36.984433 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7"] Nov 03 22:54:36.984969 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:36.984671 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" podUID="77d251f7-a96d-410e-9c90-507c492aee37" containerName="maas-api" containerID="cri-o://342c59d13806de580d4d80d24bf559191a7bdf5b35333ab7a15c7f74a477c16c" gracePeriod=30 Nov 03 22:54:37.189905 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.189873 4625 generic.go:356] "Generic (PLEG): container finished" podID="77d251f7-a96d-410e-9c90-507c492aee37" containerID="342c59d13806de580d4d80d24bf559191a7bdf5b35333ab7a15c7f74a477c16c" exitCode=0 Nov 03 22:54:37.190293 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.189987 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" event={"ID":"77d251f7-a96d-410e-9c90-507c492aee37","Type":"ContainerDied","Data":"342c59d13806de580d4d80d24bf559191a7bdf5b35333ab7a15c7f74a477c16c"} Nov 03 22:54:37.446497 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.446469 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:54:37.490454 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.489165 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d251f7-a96d-410e-9c90-507c492aee37-kube-api-access-scxb4\" (UniqueName: \"kubernetes.io/projected/77d251f7-a96d-410e-9c90-507c492aee37-kube-api-access-scxb4\") pod \"77d251f7-a96d-410e-9c90-507c492aee37\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " Nov 03 22:54:37.490454 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.489255 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-maas-api-tls\") pod \"77d251f7-a96d-410e-9c90-507c492aee37\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " Nov 03 22:54:37.490454 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.489419 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-metrics-certs\") pod \"77d251f7-a96d-410e-9c90-507c492aee37\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " Nov 03 22:54:37.490454 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.489520 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d251f7-a96d-410e-9c90-507c492aee37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d251f7-a96d-410e-9c90-507c492aee37-openshift-service-ca\") pod \"77d251f7-a96d-410e-9c90-507c492aee37\" (UID: \"77d251f7-a96d-410e-9c90-507c492aee37\") " Nov 03 22:54:37.491427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.490970 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d251f7-a96d-410e-9c90-507c492aee37-openshift-service-ca" pod "77d251f7-a96d-410e-9c90-507c492aee37" (UID: "77d251f7-a96d-410e-9c90-507c492aee37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:37.518382 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.516827 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-maas-api-tls" pod "77d251f7-a96d-410e-9c90-507c492aee37" (UID: "77d251f7-a96d-410e-9c90-507c492aee37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:37.518382 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.517065 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d251f7-a96d-410e-9c90-507c492aee37-kube-api-access-scxb4" pod "77d251f7-a96d-410e-9c90-507c492aee37" (UID: "77d251f7-a96d-410e-9c90-507c492aee37"). InnerVolumeSpecName "kube-api-access-scxb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:37.518382 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.518325 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-metrics-certs" pod "77d251f7-a96d-410e-9c90-507c492aee37" (UID: "77d251f7-a96d-410e-9c90-507c492aee37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:37.592024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.591985 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d251f7-a96d-410e-9c90-507c492aee37-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:37.592024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.592017 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxb4\" (UniqueName: \"kubernetes.io/projected/77d251f7-a96d-410e-9c90-507c492aee37-kube-api-access-scxb4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:37.592259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.592031 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:37.592259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:37.592043 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d251f7-a96d-410e-9c90-507c492aee37-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:38.198810 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:38.198754 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" event={"ID":"77d251f7-a96d-410e-9c90-507c492aee37","Type":"ContainerDied","Data":"df263a71a4e0fc1a21d30fc9644de5f7307347c072924a813d39603479c4d3d8"} Nov 03 22:54:38.199270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:38.198829 4625 scope.go:122] "RemoveContainer" containerID="342c59d13806de580d4d80d24bf559191a7bdf5b35333ab7a15c7f74a477c16c" Nov 03 22:54:38.199270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:38.198785 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7" Nov 03 22:54:38.252337 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:38.252308 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7"] Nov 03 22:54:38.265582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:38.265547 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb1fa5-6bdd7bc9d9-86vg7"] Nov 03 22:54:38.730536 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:38.730502 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d251f7-a96d-410e-9c90-507c492aee37" path="/var/lib/kubelet/pods/77d251f7-a96d-410e-9c90-507c492aee37/volumes" Nov 03 22:54:40.046766 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.046739 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:54:40.129615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.129581 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-credential-socket\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.129830 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.129645 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-kube-api-access-9b5jh\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-kube-api-access-9b5jh\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.129904 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.129855 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istiod-ca-cert\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.129993 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.129963 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-credential-socket" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:40.130224 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130205 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istiod-ca-cert" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:40.130309 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130252 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-data\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.130309 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130276 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-podinfo\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.130309 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130303 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-envoy\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.130477 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130356 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-certs\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.130534 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130515 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-data" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:40.130956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130599 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-socket\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.130956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130649 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-token\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-token\") pod \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\" (UID: \"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7\") " Nov 03 22:54:40.130956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130871 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-certs" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:40.130956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.130928 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-socket" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:40.131303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.131285 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.131379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.131307 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.131379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.131321 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.131379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.131333 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.131379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.131346 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.132502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.132474 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-kube-api-access-9b5jh" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "kube-api-access-9b5jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:40.139075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.133372 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-podinfo" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:40.139075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.133448 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-token" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:40.140073 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.140040 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-envoy" pod "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" (UID: "3d4eade4-f7c3-4160-a9c4-2c23dfb73db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:40.220636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.220603 4625 generic.go:356] "Generic (PLEG): container finished" podID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerID="686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212" exitCode=0 Nov 03 22:54:40.220885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.220653 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" Nov 03 22:54:40.220885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.220681 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" event={"ID":"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7","Type":"ContainerDied","Data":"686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212"} Nov 03 22:54:40.220885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.220771 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc" event={"ID":"3d4eade4-f7c3-4160-a9c4-2c23dfb73db7","Type":"ContainerDied","Data":"431916a897042ccc9e1fb3c967825ef1f1bab518990b21bc48cd1b22ac9d5c63"} Nov 03 22:54:40.220885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.220795 4625 scope.go:122] "RemoveContainer" containerID="686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212" Nov 03 22:54:40.232332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.232315 4625 scope.go:122] "RemoveContainer" containerID="686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212" Nov 03 22:54:40.232664 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:54:40.232646 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212\": container with ID starting with 686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212 not found: ID does not exist" containerID="686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212" Nov 03 22:54:40.232756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.232676 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212"} err="failed to get container status \"686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212\": rpc error: code = NotFound desc = could not find container \"686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212\": container with ID starting with 686d7f62fbecd3b3538bd43dc30dfc4a1ac858de42db18fef1b50f23eb4d0212 not found: ID does not exist" Nov 03 22:54:40.233002 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.232985 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.233053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.233005 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.233053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.233014 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.233053 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.233021 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b5jh\" (UniqueName: \"kubernetes.io/projected/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7-kube-api-access-9b5jh\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.278653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.278578 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc"] Nov 03 22:54:40.284052 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.284028 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-84rpc"] Nov 03 22:54:40.728685 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:40.728652 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" path="/var/lib/kubelet/pods/3d4eade4-f7c3-4160-a9c4-2c23dfb73db7/volumes" Nov 03 22:54:41.632112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:41.632077 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:41.645455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:41.645243 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:42.251184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:42.251150 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 22:54:49.389790 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.389757 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns"] Nov 03 22:54:49.390300 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.389976 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" podUID="bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" containerName="maas-api" containerID="cri-o://0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339" gracePeriod=30 Nov 03 22:54:49.731603 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.731572 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:54:49.828741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.811340 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-openshift-service-ca\") pod \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " Nov 03 22:54:49.828741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.811424 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-maas-api-tls\") pod \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " Nov 03 22:54:49.828741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.811507 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-metrics-certs\") pod \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " Nov 03 22:54:49.828741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.811611 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-kube-api-access-94kfv\" (UniqueName: \"kubernetes.io/projected/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-kube-api-access-94kfv\") pod \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\" (UID: \"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35\") " Nov 03 22:54:49.832599 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.832562 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-openshift-service-ca" pod "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" (UID: "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:49.853264 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.853217 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-kube-api-access-94kfv" pod "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" (UID: "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35"). InnerVolumeSpecName "kube-api-access-94kfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:49.855207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.855166 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-metrics-certs" pod "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" (UID: "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:49.864490 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.864406 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-maas-api-tls" pod "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" (UID: "bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:49.918642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.918606 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.918642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.918634 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94kfv\" (UniqueName: \"kubernetes.io/projected/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-kube-api-access-94kfv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.918642 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.918647 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.918992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:49.918655 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:54:50.307072 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.307040 4625 generic.go:356] "Generic (PLEG): container finished" podID="bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" containerID="0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339" exitCode=0 Nov 03 22:54:50.307262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.307087 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" Nov 03 22:54:50.307262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.307123 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" event={"ID":"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35","Type":"ContainerDied","Data":"0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339"} Nov 03 22:54:50.307262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.307182 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns" event={"ID":"bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35","Type":"ContainerDied","Data":"150b215b11025d2ce58940989e19f20ac8f95fa613b8c6f4895716ccbccdcbb8"} Nov 03 22:54:50.307262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.307213 4625 scope.go:122] "RemoveContainer" containerID="0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339" Nov 03 22:54:50.318849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.318833 4625 scope.go:122] "RemoveContainer" containerID="0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339" Nov 03 22:54:50.319113 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:54:50.319087 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339\": container with ID starting with 0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339 not found: ID does not exist" containerID="0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339" Nov 03 22:54:50.319174 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.319127 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339"} err="failed to get container status \"0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339\": rpc error: code = NotFound desc = could not find container \"0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339\": container with ID starting with 0a09f9c5a752fc4fcaf9ee3ea9395ae10a34e6861ca0d7d99e6e738c1e412339 not found: ID does not exist" Nov 03 22:54:50.362820 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.362787 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns"] Nov 03 22:54:50.374274 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.374246 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-124490-67cf698669-kx9ns"] Nov 03 22:54:50.728666 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:54:50.728638 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" path="/var/lib/kubelet/pods/bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35/volumes" Nov 03 22:55:02.187331 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:55:02.187289 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:55:02.519972 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.519880 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz"] Nov 03 22:55:02.520868 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.520846 4625 cpu_manager.go:414] "containerMap: removing container" podUID="bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" containerName="maas-api" Nov 03 22:55:02.520868 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.520867 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" containerName="maas-api" Nov 03 22:55:02.521071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.520887 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" Nov 03 22:55:02.521071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.520895 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" Nov 03 22:55:02.521071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.520941 4625 cpu_manager.go:414] "containerMap: removing container" podUID="77d251f7-a96d-410e-9c90-507c492aee37" containerName="maas-api" Nov 03 22:55:02.521071 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.520949 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d251f7-a96d-410e-9c90-507c492aee37" containerName="maas-api" Nov 03 22:55:02.521283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.521092 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d251f7-a96d-410e-9c90-507c492aee37" containerName="maas-api" Nov 03 22:55:02.521283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.521112 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb37d5be-770b-46cf-a9d7-ec7d4c6c4b35" containerName="maas-api" Nov 03 22:55:02.521283 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.521122 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4eade4-f7c3-4160-a9c4-2c23dfb73db7" containerName="istio-proxy" Nov 03 22:55:02.525482 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.525435 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.541801 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.541774 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz"] Nov 03 22:55:02.547480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.547437 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-0847aa\"/\"default-dockercfg-bh6fw\"" Nov 03 22:55:02.555346 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.555320 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-0847aa\"/\"sub-test-model-0847aa-kserve-self-signed-certs\"" Nov 03 22:55:02.555483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.555333 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-0847aa\"/\"kube-root-ca.crt\"" Nov 03 22:55:02.555483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.555437 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-0847aa\"/\"openshift-service-ca.crt\"" Nov 03 22:55:02.624136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.624105 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tmp-dir\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.624136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.624137 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-model-cache\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.624344 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.624220 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-dshm\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.624388 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.624358 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ddw\" (UniqueName: \"kubernetes.io/projected/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-kube-api-access-w8ddw\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.624422 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.624391 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tls-certs\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.624561 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.624548 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-home\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.725292 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.725263 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-model-cache\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.725468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.725322 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-dshm\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.725468 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.725448 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ddw\" (UniqueName: \"kubernetes.io/projected/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-kube-api-access-w8ddw\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.725992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.725483 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tls-certs\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.725992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.725810 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-home\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.725992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.725817 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-model-cache\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.726409 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.726372 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tmp-dir\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.726607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.726567 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-home\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.726702 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.726689 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tmp-dir\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.728269 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.728247 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-dshm\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.728929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.728909 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tls-certs\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.748372 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.748344 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ddw\" (UniqueName: \"kubernetes.io/projected/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-kube-api-access-w8ddw\") pod \"sub-test-model-0847aa-kserve-55469f7855-nq2fz\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:02.848101 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:02.848054 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:03.184926 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:03.184811 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz"] Nov 03 22:55:03.190679 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:55:03.190649 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdc38cc9_0fa8_4546_86e0_1c93f3872a2f.slice/crio-8ea74383df775b18aded6728b1d7701aab2a82171df392398ab98353ad3e92ec WatchSource:0}: Error finding container 8ea74383df775b18aded6728b1d7701aab2a82171df392398ab98353ad3e92ec: Status 404 returned error can't find the container with id 8ea74383df775b18aded6728b1d7701aab2a82171df392398ab98353ad3e92ec Nov 03 22:55:03.442093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:03.441999 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" event={"ID":"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f","Type":"ContainerStarted","Data":"ad8b0b40a3339ff92d76919ce64bbf980a2da1b3ad0a967acfef231ca75e9e4f"} Nov 03 22:55:03.442093 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:03.442048 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" event={"ID":"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f","Type":"ContainerStarted","Data":"8ea74383df775b18aded6728b1d7701aab2a82171df392398ab98353ad3e92ec"} Nov 03 22:55:03.483028 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:03.482968 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" podStartSLOduration=1.482952517 podStartE2EDuration="1.482952517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:03.481967176 +0000 UTC m=+2524.911303169" watchObservedRunningTime="2026-11-03 22:55:03.482952517 +0000 UTC m=+2524.912288499" Nov 03 22:55:03.851985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:03.851951 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:12.852954 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:12.852914 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:12.867374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:12.867344 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:13.533330 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:13.533297 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 22:55:16.147139 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.147101 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2"] Nov 03 22:55:16.147628 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.147363 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" podUID="34857ca0-8bb7-4b57-a795-35aaa7c4ba65" containerName="maas-api" containerID="cri-o://dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a" gracePeriod=30 Nov 03 22:55:16.594962 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.594621 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:55:16.606759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.604120 4625 generic.go:356] "Generic (PLEG): container finished" podID="34857ca0-8bb7-4b57-a795-35aaa7c4ba65" containerID="dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a" exitCode=0 Nov 03 22:55:16.606759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.604222 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" event={"ID":"34857ca0-8bb7-4b57-a795-35aaa7c4ba65","Type":"ContainerDied","Data":"dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a"} Nov 03 22:55:16.606759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.604271 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" event={"ID":"34857ca0-8bb7-4b57-a795-35aaa7c4ba65","Type":"ContainerDied","Data":"5d736094788d944953e8d10b5f5cf0ab03e978b0c068cb6c75dc900529991ab9"} Nov 03 22:55:16.606759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.604299 4625 scope.go:122] "RemoveContainer" containerID="dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a" Nov 03 22:55:16.606759 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.604451 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2" Nov 03 22:55:16.649044 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.649023 4625 scope.go:122] "RemoveContainer" containerID="dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a" Nov 03 22:55:16.649581 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:55:16.649555 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a\": container with ID starting with dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a not found: ID does not exist" containerID="dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a" Nov 03 22:55:16.649704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.649686 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a"} err="failed to get container status \"dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a\": rpc error: code = NotFound desc = could not find container \"dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a\": container with ID starting with dbac7e675952e42f0cc2605e5d390279a35bbc5ffe1e76f46fa750713946a78a not found: ID does not exist" Nov 03 22:55:16.773726 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.773616 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-kube-api-access-k5m79\" (UniqueName: \"kubernetes.io/projected/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-kube-api-access-k5m79\") pod \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " Nov 03 22:55:16.774112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.774046 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-openshift-service-ca" pod "34857ca0-8bb7-4b57-a795-35aaa7c4ba65" (UID: "34857ca0-8bb7-4b57-a795-35aaa7c4ba65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:16.774112 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.774092 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-openshift-service-ca\") pod \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " Nov 03 22:55:16.774324 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.774180 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls\") pod \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " Nov 03 22:55:16.774405 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.774390 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-metrics-certs\") pod \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\" (UID: \"34857ca0-8bb7-4b57-a795-35aaa7c4ba65\") " Nov 03 22:55:16.775331 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.775303 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:55:16.775897 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.775863 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-kube-api-access-k5m79" pod "34857ca0-8bb7-4b57-a795-35aaa7c4ba65" (UID: "34857ca0-8bb7-4b57-a795-35aaa7c4ba65"). InnerVolumeSpecName "kube-api-access-k5m79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:16.776358 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.776333 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls" pod "34857ca0-8bb7-4b57-a795-35aaa7c4ba65" (UID: "34857ca0-8bb7-4b57-a795-35aaa7c4ba65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:55:16.776756 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.776735 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-metrics-certs" pod "34857ca0-8bb7-4b57-a795-35aaa7c4ba65" (UID: "34857ca0-8bb7-4b57-a795-35aaa7c4ba65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:55:16.876881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.876850 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:55:16.876881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.876881 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:55:16.877096 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.876894 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5m79\" (UniqueName: \"kubernetes.io/projected/34857ca0-8bb7-4b57-a795-35aaa7c4ba65-kube-api-access-k5m79\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:55:16.965129 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.965082 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2"] Nov 03 22:55:16.982225 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:16.979893 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963936-5f5c97895c-sctt2"] Nov 03 22:55:18.730822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:18.730785 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34857ca0-8bb7-4b57-a795-35aaa7c4ba65" path="/var/lib/kubelet/pods/34857ca0-8bb7-4b57-a795-35aaa7c4ba65/volumes" Nov 03 22:55:25.469331 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.469290 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4"] Nov 03 22:55:25.470342 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.470319 4625 cpu_manager.go:414] "containerMap: removing container" podUID="34857ca0-8bb7-4b57-a795-35aaa7c4ba65" containerName="maas-api" Nov 03 22:55:25.470455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.470343 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34857ca0-8bb7-4b57-a795-35aaa7c4ba65" containerName="maas-api" Nov 03 22:55:25.470602 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.470588 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="34857ca0-8bb7-4b57-a795-35aaa7c4ba65" containerName="maas-api" Nov 03 22:55:25.476488 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.476459 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.497705 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.497673 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-68410c-openshift-default-dockercfg-z8dwq\"" Nov 03 22:55:25.522240 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522201 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-data\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522331 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-socket\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522361 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-certs\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522511 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522492 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzdr\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-kube-api-access-5dzdr\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522552 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522524 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-envoy\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522597 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522573 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-credential-socket\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522636 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522598 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-token\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522674 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522643 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-podinfo\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.522742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.522708 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b89cf6c-f171-4e05-a770-63e6d790eef8-istiod-ca-cert\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.527955 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.527921 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4"] Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637113 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b89cf6c-f171-4e05-a770-63e6d790eef8-istiod-ca-cert\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637212 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-data\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637314 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-socket\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637339 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-certs\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637462 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzdr\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-kube-api-access-5dzdr\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637486 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-envoy\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637530 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-credential-socket\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637551 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-token\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.637595 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-podinfo\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.638147 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-certs\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.638811 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b89cf6c-f171-4e05-a770-63e6d790eef8-istiod-ca-cert\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.639076 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-data\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.639303 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-socket\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.640741 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.639901 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-credential-socket\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.659087 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.659051 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-envoy\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.698751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.677474 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-token\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.698751 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.686393 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-podinfo\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.713740 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.702999 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzdr\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-kube-api-access-5dzdr\") pod \"e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:25.793625 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:25.793549 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:26.154095 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.151480 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4"] Nov 03 22:55:26.157610 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:55:26.157579 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b89cf6c_f171_4e05_a770_63e6d790eef8.slice/crio-134b1829df192c635b3d8e65f9db59bf80584c84fff758cc09af0f77a324702b WatchSource:0}: Error finding container 134b1829df192c635b3d8e65f9db59bf80584c84fff758cc09af0f77a324702b: Status 404 returned error can't find the container with id 134b1829df192c635b3d8e65f9db59bf80584c84fff758cc09af0f77a324702b Nov 03 22:55:26.161429 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.161390 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:55:26.161530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.161481 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:55:26.161530 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.161523 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:55:26.714451 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.714418 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" event={"ID":"1b89cf6c-f171-4e05-a770-63e6d790eef8","Type":"ContainerStarted","Data":"8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc"} Nov 03 22:55:26.714913 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.714470 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" event={"ID":"1b89cf6c-f171-4e05-a770-63e6d790eef8","Type":"ContainerStarted","Data":"134b1829df192c635b3d8e65f9db59bf80584c84fff758cc09af0f77a324702b"} Nov 03 22:55:26.770891 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.770839 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podStartSLOduration=1.770824655 podStartE2EDuration="1.770824655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:26.758578624 +0000 UTC m=+2548.187914616" watchObservedRunningTime="2026-11-03 22:55:26.770824655 +0000 UTC m=+2548.200160645" Nov 03 22:55:26.794370 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:26.794343 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:27.800333 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:27.800305 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:28.732314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:28.732285 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:28.733583 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:55:28.733555 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:55:32.188047 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:55:32.187993 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:56:01.554614 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.554582 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d"] Nov 03 22:56:01.559244 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.559222 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.584279 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.584241 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-68410c-serving-cert\"" Nov 03 22:56:01.591136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.591099 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d"] Nov 03 22:56:01.663213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.663173 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpv4\" (UniqueName: \"kubernetes.io/projected/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-kube-api-access-hqpv4\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.663393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.663284 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.663442 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.663392 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.663505 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.663487 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.764676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.764637 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.765909 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.765883 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.766397 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.766374 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpv4\" (UniqueName: \"kubernetes.io/projected/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-kube-api-access-hqpv4\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.766589 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.766574 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.767515 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.767494 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.771624 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.771575 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.818374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.804941 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.818374 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.810403 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpv4\" (UniqueName: \"kubernetes.io/projected/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-kube-api-access-hqpv4\") pod \"maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:01.874885 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:01.874850 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:02.185392 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:02.185361 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d"] Nov 03 22:56:02.186678 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:56:02.186647 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f1b9f9_0bd1_48fe_841e_8d51d17db9ba.slice/crio-8e51f6c81172fa369b241a0d27663072f24a116dc4086b50931a3add80bb7314 WatchSource:0}: Error finding container 8e51f6c81172fa369b241a0d27663072f24a116dc4086b50931a3add80bb7314: Status 404 returned error can't find the container with id 8e51f6c81172fa369b241a0d27663072f24a116dc4086b50931a3add80bb7314 Nov 03 22:56:02.188313 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:56:02.188286 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:56:02.188529 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:02.188513 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:56:03.082535 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:03.082496 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" event={"ID":"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba","Type":"ContainerStarted","Data":"99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab"} Nov 03 22:56:03.083022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:03.082557 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" event={"ID":"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba","Type":"ContainerStarted","Data":"8e51f6c81172fa369b241a0d27663072f24a116dc4086b50931a3add80bb7314"} Nov 03 22:56:03.083022 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:03.082579 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:03.124314 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:03.124260 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" podStartSLOduration=1.773668585 podStartE2EDuration="2.124246579s" totalImagesPullingTime="350.577994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:03.123759115 +0000 UTC m=+2584.553095106" watchObservedRunningTime="2026-11-03 22:56:03.124246579 +0000 UTC m=+2584.553582569" Nov 03 22:56:09.096591 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:09.096553 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 22:56:29.575839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.575798 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79"] Nov 03 22:56:29.603735 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.597044 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.617106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.612255 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3920cb-serving-cert\"" Nov 03 22:56:29.617106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.614632 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79"] Nov 03 22:56:29.714136 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.714090 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7df9d1-e369-4417-888b-9e4fd6db93a7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.714331 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.714219 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-metrics-certs\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.714386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.714337 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqp9\" (UniqueName: \"kubernetes.io/projected/8a7df9d1-e369-4417-888b-9e4fd6db93a7-kube-api-access-fkqp9\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.714481 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.714464 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.815486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.815453 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.815692 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.815640 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7df9d1-e369-4417-888b-9e4fd6db93a7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.815849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.815823 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-metrics-certs\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.816011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.815944 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqp9\" (UniqueName: \"kubernetes.io/projected/8a7df9d1-e369-4417-888b-9e4fd6db93a7-kube-api-access-fkqp9\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.816391 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.816359 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7df9d1-e369-4417-888b-9e4fd6db93a7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.818097 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.818075 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.818154 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.818142 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-metrics-certs\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.898777 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.889679 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqp9\" (UniqueName: \"kubernetes.io/projected/8a7df9d1-e369-4417-888b-9e4fd6db93a7-kube-api-access-fkqp9\") pod \"maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:29.927894 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:29.927863 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:30.282870 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:30.282843 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79"] Nov 03 22:56:30.283557 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:56:30.283518 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7df9d1_e369_4417_888b_9e4fd6db93a7.slice/crio-28c8500c967ca147b52a15f365f16fd46a842d30a34b9aaa5cae5c0e8fa631a0 WatchSource:0}: Error finding container 28c8500c967ca147b52a15f365f16fd46a842d30a34b9aaa5cae5c0e8fa631a0: Status 404 returned error can't find the container with id 28c8500c967ca147b52a15f365f16fd46a842d30a34b9aaa5cae5c0e8fa631a0 Nov 03 22:56:30.327019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:30.326974 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" event={"ID":"8a7df9d1-e369-4417-888b-9e4fd6db93a7","Type":"ContainerStarted","Data":"28c8500c967ca147b52a15f365f16fd46a842d30a34b9aaa5cae5c0e8fa631a0"} Nov 03 22:56:31.374974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:31.373793 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" event={"ID":"8a7df9d1-e369-4417-888b-9e4fd6db93a7","Type":"ContainerStarted","Data":"0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c"} Nov 03 22:56:31.374974 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:31.374906 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:31.419037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:31.418990 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" podStartSLOduration=1.954007343 podStartE2EDuration="2.418976312s" totalImagesPullingTime="464.968969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:31.415270589 +0000 UTC m=+2612.844606579" watchObservedRunningTime="2026-11-03 22:56:31.418976312 +0000 UTC m=+2612.848312303" Nov 03 22:56:32.195035 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:56:32.194991 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:56:37.423767 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:37.423703 4625 generic.go:356] "Generic (PLEG): container finished" podID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerID="8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc" exitCode=0 Nov 03 22:56:37.424218 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:37.423774 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" event={"ID":"1b89cf6c-f171-4e05-a770-63e6d790eef8","Type":"ContainerDied","Data":"8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc"} Nov 03 22:56:37.424263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:37.424220 4625 scope.go:122] "RemoveContainer" containerID="8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc" Nov 03 22:56:37.425942 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:37.425909 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:56:37.426016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:37.425979 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:56:37.426016 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:37.426007 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:56:38.397733 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:38.397695 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 22:56:38.434054 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:38.434012 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" event={"ID":"1b89cf6c-f171-4e05-a770-63e6d790eef8","Type":"ContainerStarted","Data":"6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9"} Nov 03 22:56:38.794107 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:38.794021 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:56:38.794303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:38.794279 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 22:56:38.794362 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:38.794328 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 22:56:39.794207 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:39.794172 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 22:56:39.794643 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:39.794224 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 22:56:40.794883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:40.794845 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 22:56:40.795366 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:40.794911 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 22:56:41.794780 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:41.794700 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 22:56:41.794977 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:41.794824 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 22:56:42.795029 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:42.794985 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 22:56:42.795532 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:42.795052 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 22:56:43.732402 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:43.732367 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:56:43.794251 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:43.794218 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 22:56:43.794423 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:43.794271 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 22:56:44.799944 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:44.799911 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:56:44.800923 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:56:44.800905 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 22:57:02.185598 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:57:02.185563 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:57:32.186868 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:57:32.186790 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:57:44.621343 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:44.621309 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q"] Nov 03 22:57:56.799024 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.798991 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk"] Nov 03 22:57:56.804486 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.804459 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.824162 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.821592 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f48f46-openshift-default-dockercfg-6qdf5\"" Nov 03 22:57:56.842675 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.842642 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk"] Nov 03 22:57:56.907699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.907662 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25766c62-3e88-493d-bfeb-a3e33b86e67c-istiod-ca-cert\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.907976 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.907955 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-podinfo\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.908198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.908182 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-certs\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.908329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.908315 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-envoy\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.908542 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.908525 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wz7\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-kube-api-access-77wz7\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.908667 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.908648 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-token\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.908822 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.908802 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-data\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.909223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.908987 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-credential-socket\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:56.909223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:56.909094 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-socket\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.012665 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wz7\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-kube-api-access-77wz7\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.012735 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-token\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.012765 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-data\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.012883 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-credential-socket\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.012947 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-socket\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.012994 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25766c62-3e88-493d-bfeb-a3e33b86e67c-istiod-ca-cert\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.013030 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-podinfo\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.013131 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-certs\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.022738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.013159 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-envoy\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.023738 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-envoy\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.024522 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-data\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.024800 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-credential-socket\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.025029 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-socket\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.025625 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25766c62-3e88-493d-bfeb-a3e33b86e67c-istiod-ca-cert\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.039320 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-podinfo\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.039613 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-certs\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.060852 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.050286 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wz7\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-kube-api-access-77wz7\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.091664 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.071523 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-token\") pod \"e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.124653 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.124382 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:57.458629 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:57:57.458597 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25766c62_3e88_493d_bfeb_a3e33b86e67c.slice/crio-5936245e921c30e26e74ccfc7945ce918c3f47b20e4bedc8023cd3e230eca996 WatchSource:0}: Error finding container 5936245e921c30e26e74ccfc7945ce918c3f47b20e4bedc8023cd3e230eca996: Status 404 returned error can't find the container with id 5936245e921c30e26e74ccfc7945ce918c3f47b20e4bedc8023cd3e230eca996 Nov 03 22:57:57.459907 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.459882 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk"] Nov 03 22:57:57.461633 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.461601 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:57:57.461782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.461678 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:57:57.461782 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:57.461761 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:57:58.336831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:58.336786 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" event={"ID":"25766c62-3e88-493d-bfeb-a3e33b86e67c","Type":"ContainerStarted","Data":"6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd"} Nov 03 22:57:58.337270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:58.336893 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" event={"ID":"25766c62-3e88-493d-bfeb-a3e33b86e67c","Type":"ContainerStarted","Data":"5936245e921c30e26e74ccfc7945ce918c3f47b20e4bedc8023cd3e230eca996"} Nov 03 22:57:58.407098 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:58.407040 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" podStartSLOduration=2.4070261889999998 podStartE2EDuration="2.407026189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:58.391982717 +0000 UTC m=+2699.821318709" watchObservedRunningTime="2026-11-03 22:57:58.407026189 +0000 UTC m=+2699.836362238" Nov 03 22:57:59.125439 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:59.125407 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:59.130861 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:59.130835 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:59.346242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:59.346212 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:59.347286 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:59.347264 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:57:59.646065 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:59.646027 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" podUID="6076e4a0-c1c0-4691-b20c-0a163396385b" containerName="main" containerID="cri-o://ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198" gracePeriod=45 Nov 03 22:57:59.725046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:59.725006 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" podUID="6076e4a0-c1c0-4691-b20c-0a163396385b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.177:8000/ready\": dial tcp 10.134.0.177:8000: connect: connection refused" Nov 03 22:57:59.982331 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:57:59.982298 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:58:00.091114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091084 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-model-cache\") pod \"6076e4a0-c1c0-4691-b20c-0a163396385b\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " Nov 03 22:58:00.091321 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091133 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-tmp-dir\") pod \"6076e4a0-c1c0-4691-b20c-0a163396385b\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " Nov 03 22:58:00.091321 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091264 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-home\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-home\") pod \"6076e4a0-c1c0-4691-b20c-0a163396385b\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " Nov 03 22:58:00.091431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091343 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-model-cache" pod "6076e4a0-c1c0-4691-b20c-0a163396385b" (UID: "6076e4a0-c1c0-4691-b20c-0a163396385b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:00.091431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091363 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6076e4a0-c1c0-4691-b20c-0a163396385b-kube-api-access-nw8zg\" (UniqueName: \"kubernetes.io/projected/6076e4a0-c1c0-4691-b20c-0a163396385b-kube-api-access-nw8zg\") pod \"6076e4a0-c1c0-4691-b20c-0a163396385b\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " Nov 03 22:58:00.091431 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091425 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-dshm\") pod \"6076e4a0-c1c0-4691-b20c-0a163396385b\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " Nov 03 22:58:00.091550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091426 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-tmp-dir" pod "6076e4a0-c1c0-4691-b20c-0a163396385b" (UID: "6076e4a0-c1c0-4691-b20c-0a163396385b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:00.091550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091466 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6076e4a0-c1c0-4691-b20c-0a163396385b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6076e4a0-c1c0-4691-b20c-0a163396385b-tls-certs\") pod \"6076e4a0-c1c0-4691-b20c-0a163396385b\" (UID: \"6076e4a0-c1c0-4691-b20c-0a163396385b\") " Nov 03 22:58:00.091550 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.091497 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-home" pod "6076e4a0-c1c0-4691-b20c-0a163396385b" (UID: "6076e4a0-c1c0-4691-b20c-0a163396385b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:00.092037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.092014 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:00.092037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.092033 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:00.092037 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.092041 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:00.093608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.093582 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6076e4a0-c1c0-4691-b20c-0a163396385b-tls-certs" pod "6076e4a0-c1c0-4691-b20c-0a163396385b" (UID: "6076e4a0-c1c0-4691-b20c-0a163396385b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:00.093608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.093598 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-dshm" pod "6076e4a0-c1c0-4691-b20c-0a163396385b" (UID: "6076e4a0-c1c0-4691-b20c-0a163396385b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:00.093788 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.093656 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6076e4a0-c1c0-4691-b20c-0a163396385b-kube-api-access-nw8zg" pod "6076e4a0-c1c0-4691-b20c-0a163396385b" (UID: "6076e4a0-c1c0-4691-b20c-0a163396385b"). InnerVolumeSpecName "kube-api-access-nw8zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:00.193615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.193520 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6076e4a0-c1c0-4691-b20c-0a163396385b-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:00.193615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.193550 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6076e4a0-c1c0-4691-b20c-0a163396385b-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:00.193615 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.193563 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8zg\" (UniqueName: \"kubernetes.io/projected/6076e4a0-c1c0-4691-b20c-0a163396385b-kube-api-access-nw8zg\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:00.356213 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.356175 4625 generic.go:356] "Generic (PLEG): container finished" podID="6076e4a0-c1c0-4691-b20c-0a163396385b" containerID="ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198" exitCode=0 Nov 03 22:58:00.356750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.356219 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" Nov 03 22:58:00.356750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.356249 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" event={"ID":"6076e4a0-c1c0-4691-b20c-0a163396385b","Type":"ContainerDied","Data":"ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198"} Nov 03 22:58:00.356750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.356302 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q" event={"ID":"6076e4a0-c1c0-4691-b20c-0a163396385b","Type":"ContainerDied","Data":"45e4de727d5cb863604a2083c274cddb6f90db5df98cf11d60334f7eca3bd535"} Nov 03 22:58:00.356750 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.356330 4625 scope.go:122] "RemoveContainer" containerID="ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198" Nov 03 22:58:00.369872 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.369854 4625 scope.go:122] "RemoveContainer" containerID="ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198" Nov 03 22:58:00.370159 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:58:00.370126 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198\": container with ID starting with ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198 not found: ID does not exist" containerID="ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198" Nov 03 22:58:00.370262 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.370184 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198"} err="failed to get container status \"ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198\": rpc error: code = NotFound desc = could not find container \"ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198\": container with ID starting with ce475c773bb65b90106f9c831a363c5643c29431668e51f31ac8b24ae22cd198 not found: ID does not exist" Nov 03 22:58:00.411654 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.411626 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q"] Nov 03 22:58:00.415650 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.415619 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-d4b6bb/auth-test-model-d4b6bb-kserve-6499d64885-6x92q"] Nov 03 22:58:00.729329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:00.729301 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6076e4a0-c1c0-4691-b20c-0a163396385b" path="/var/lib/kubelet/pods/6076e4a0-c1c0-4691-b20c-0a163396385b/volumes" Nov 03 22:58:02.187467 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:58:02.187426 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:58:03.869427 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:03.869402 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:58:03.916965 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:03.916936 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 22:58:07.931884 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:07.931854 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc"] Nov 03 22:58:07.932544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:07.932529 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6076e4a0-c1c0-4691-b20c-0a163396385b" containerName="main" Nov 03 22:58:07.932544 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:07.932544 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6076e4a0-c1c0-4691-b20c-0a163396385b" containerName="main" Nov 03 22:58:07.932703 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:07.932689 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6076e4a0-c1c0-4691-b20c-0a163396385b" containerName="main" Nov 03 22:58:07.938089 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:07.938051 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:07.957522 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:07.957495 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ff80c0-serving-cert\"" Nov 03 22:58:07.960233 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:07.960192 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc"] Nov 03 22:58:08.039219 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.039186 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-metrics-certs\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.039388 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.039252 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.039445 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.039377 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.039524 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.039509 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jlm\" (UniqueName: \"kubernetes.io/projected/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-kube-api-access-s9jlm\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.148878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.143313 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.148878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.143463 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jlm\" (UniqueName: \"kubernetes.io/projected/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-kube-api-access-s9jlm\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.148878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.143622 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-metrics-certs\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.148878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.143689 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.148878 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.144600 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.163971 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.163835 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-metrics-certs\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.164981 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.164954 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.169956 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.169929 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jlm\" (UniqueName: \"kubernetes.io/projected/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-kube-api-access-s9jlm\") pod \"maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.260760 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.260639 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:08.576502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:08.576466 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc"] Nov 03 22:58:08.579095 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:58:08.579064 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded89659f_e1b0_4c90_b1e8_35a8518fc4cf.slice/crio-dc94c1882c22ee4898d455aac386f8d0382edef12430835ee4319703e764c137 WatchSource:0}: Error finding container dc94c1882c22ee4898d455aac386f8d0382edef12430835ee4319703e764c137: Status 404 returned error can't find the container with id dc94c1882c22ee4898d455aac386f8d0382edef12430835ee4319703e764c137 Nov 03 22:58:09.482168 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:09.482126 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" event={"ID":"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf","Type":"ContainerStarted","Data":"bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1"} Nov 03 22:58:09.482555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:09.482186 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" event={"ID":"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf","Type":"ContainerStarted","Data":"dc94c1882c22ee4898d455aac386f8d0382edef12430835ee4319703e764c137"} Nov 03 22:58:09.482555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:09.482214 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:09.529319 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:09.529269 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" podStartSLOduration=2.138834415 podStartE2EDuration="2.529254528s" totalImagesPullingTime="390.420113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:09.528134022 +0000 UTC m=+2710.957470009" watchObservedRunningTime="2026-11-03 22:58:09.529254528 +0000 UTC m=+2710.958590519" Nov 03 22:58:12.506003 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:12.505966 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r"] Nov 03 22:58:15.517989 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:15.517961 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 22:58:27.532411 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.532353 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" podUID="7bee8ce9-e7a9-4119-a3e5-87913b61099d" containerName="main" containerID="cri-o://1c33535da131eaedf82d2afcc62ab37eaf97dc88b1705edcb95acbba9e214463" gracePeriod=45 Nov 03 22:58:27.665681 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.665649 4625 generic.go:356] "Generic (PLEG): container finished" podID="7bee8ce9-e7a9-4119-a3e5-87913b61099d" containerID="1c33535da131eaedf82d2afcc62ab37eaf97dc88b1705edcb95acbba9e214463" exitCode=0 Nov 03 22:58:27.665883 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.665815 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" event={"ID":"7bee8ce9-e7a9-4119-a3e5-87913b61099d","Type":"ContainerDied","Data":"1c33535da131eaedf82d2afcc62ab37eaf97dc88b1705edcb95acbba9e214463"} Nov 03 22:58:27.812210 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.812183 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:58:27.895137 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895099 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tmp-dir\") pod \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " Nov 03 22:58:27.895338 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895212 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tls-certs\") pod \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " Nov 03 22:58:27.895338 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895290 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-dshm\") pod \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " Nov 03 22:58:27.895470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895370 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tmp-dir" pod "7bee8ce9-e7a9-4119-a3e5-87913b61099d" (UID: "7bee8ce9-e7a9-4119-a3e5-87913b61099d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:27.895470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895408 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-home\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-home\") pod \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " Nov 03 22:58:27.895470 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895449 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-model-cache\") pod \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " Nov 03 22:58:27.895644 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895491 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bee8ce9-e7a9-4119-a3e5-87913b61099d-kube-api-access-tlzr6\" (UniqueName: \"kubernetes.io/projected/7bee8ce9-e7a9-4119-a3e5-87913b61099d-kube-api-access-tlzr6\") pod \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\" (UID: \"7bee8ce9-e7a9-4119-a3e5-87913b61099d\") " Nov 03 22:58:27.896023 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.895994 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-model-cache" pod "7bee8ce9-e7a9-4119-a3e5-87913b61099d" (UID: "7bee8ce9-e7a9-4119-a3e5-87913b61099d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:27.896253 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.896232 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:27.896329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.896258 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:27.896329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.896320 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-home" pod "7bee8ce9-e7a9-4119-a3e5-87913b61099d" (UID: "7bee8ce9-e7a9-4119-a3e5-87913b61099d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:27.897608 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.897569 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tls-certs" pod "7bee8ce9-e7a9-4119-a3e5-87913b61099d" (UID: "7bee8ce9-e7a9-4119-a3e5-87913b61099d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:27.897743 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.897651 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-dshm" pod "7bee8ce9-e7a9-4119-a3e5-87913b61099d" (UID: "7bee8ce9-e7a9-4119-a3e5-87913b61099d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:27.898097 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.898070 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bee8ce9-e7a9-4119-a3e5-87913b61099d-kube-api-access-tlzr6" pod "7bee8ce9-e7a9-4119-a3e5-87913b61099d" (UID: "7bee8ce9-e7a9-4119-a3e5-87913b61099d"). InnerVolumeSpecName "kube-api-access-tlzr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:27.998106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.998073 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bee8ce9-e7a9-4119-a3e5-87913b61099d-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:27.998106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.998101 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:27.998106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.998109 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bee8ce9-e7a9-4119-a3e5-87913b61099d-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:27.998355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:27.998117 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzr6\" (UniqueName: \"kubernetes.io/projected/7bee8ce9-e7a9-4119-a3e5-87913b61099d-kube-api-access-tlzr6\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:58:28.675376 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:28.675348 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" Nov 03 22:58:28.675887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:28.675385 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r" event={"ID":"7bee8ce9-e7a9-4119-a3e5-87913b61099d","Type":"ContainerDied","Data":"780d4148340a4f7c900fae08f093ca6813f0e349e58838123cea66ec7803d164"} Nov 03 22:58:28.675887 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:28.675443 4625 scope.go:122] "RemoveContainer" containerID="1c33535da131eaedf82d2afcc62ab37eaf97dc88b1705edcb95acbba9e214463" Nov 03 22:58:28.732075 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:28.732045 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r"] Nov 03 22:58:28.741528 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:28.741497 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4f9bb0/auth-test-model-4f9bb0-kserve-f546cf865-bm29r"] Nov 03 22:58:30.728276 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:30.728242 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bee8ce9-e7a9-4119-a3e5-87913b61099d" path="/var/lib/kubelet/pods/7bee8ce9-e7a9-4119-a3e5-87913b61099d/volumes" Nov 03 22:58:32.186895 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:58:32.186856 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:58:38.256469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.256435 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j"] Nov 03 22:58:38.257455 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.257434 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7bee8ce9-e7a9-4119-a3e5-87913b61099d" containerName="main" Nov 03 22:58:38.257502 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.257458 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bee8ce9-e7a9-4119-a3e5-87913b61099d" containerName="main" Nov 03 22:58:38.257658 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.257646 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bee8ce9-e7a9-4119-a3e5-87913b61099d" containerName="main" Nov 03 22:58:38.263216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.263194 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.284040 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.284009 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f48f46-serving-cert\"" Nov 03 22:58:38.287686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.287659 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j"] Nov 03 22:58:38.327869 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.327822 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0217efd-c387-423c-8e37-a4dc7c89d36e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.328039 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.327889 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.328106 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.328093 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5vs\" (UniqueName: \"kubernetes.io/projected/e0217efd-c387-423c-8e37-a4dc7c89d36e-kube-api-access-tw5vs\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.328148 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.328128 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.429186 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.429146 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0217efd-c387-423c-8e37-a4dc7c89d36e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.429380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.429202 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.429380 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.429346 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5vs\" (UniqueName: \"kubernetes.io/projected/e0217efd-c387-423c-8e37-a4dc7c89d36e-kube-api-access-tw5vs\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.429526 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.429392 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.429932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.429902 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0217efd-c387-423c-8e37-a4dc7c89d36e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.431659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.431639 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.431798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.431780 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.452925 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.452899 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5vs\" (UniqueName: \"kubernetes.io/projected/e0217efd-c387-423c-8e37-a4dc7c89d36e-kube-api-access-tw5vs\") pod \"maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.597152 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.597114 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:38.928017 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:58:38.927986 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0217efd_c387_423c_8e37_a4dc7c89d36e.slice/crio-06613d031183734d46a68166c61e54e0f33c8642dff7694eb3de040d040c4773 WatchSource:0}: Error finding container 06613d031183734d46a68166c61e54e0f33c8642dff7694eb3de040d040c4773: Status 404 returned error can't find the container with id 06613d031183734d46a68166c61e54e0f33c8642dff7694eb3de040d040c4773 Nov 03 22:58:38.928163 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:38.928143 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j"] Nov 03 22:58:39.837555 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:39.837501 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" event={"ID":"e0217efd-c387-423c-8e37-a4dc7c89d36e","Type":"ContainerStarted","Data":"dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37"} Nov 03 22:58:39.838050 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:39.837586 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" event={"ID":"e0217efd-c387-423c-8e37-a4dc7c89d36e","Type":"ContainerStarted","Data":"06613d031183734d46a68166c61e54e0f33c8642dff7694eb3de040d040c4773"} Nov 03 22:58:39.838050 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:39.837620 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:39.884545 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:39.884480 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" podStartSLOduration=1.551130622 podStartE2EDuration="1.884466727s" totalImagesPullingTime="333.336105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:39.878896872 +0000 UTC m=+2741.308232865" watchObservedRunningTime="2026-11-03 22:58:39.884466727 +0000 UTC m=+2741.313802765" Nov 03 22:58:43.528036 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.527953 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg"] Nov 03 22:58:43.533582 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.533557 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.553607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.553574 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg"] Nov 03 22:58:43.560458 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.560428 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ff80c0-openshift-default-dockercfg-v27gw\"" Nov 03 22:58:43.622303 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622267 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622416 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622448 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622577 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622491 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622613 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622599 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622648 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622639 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622686 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622670 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622758 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622735 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcw8n\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-kube-api-access-zcw8n\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.622850 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.622832 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.723961 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.723931 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724072 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724171 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724216 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724199 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724240 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724310 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724351 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724364 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724392 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724444 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcw8n\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-kube-api-access-zcw8n\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.724683 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724568 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.725172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.724939 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.725172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.725040 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.725172 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.725136 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.726508 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.726486 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.727047 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.727030 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.747270 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.747244 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.748506 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.748473 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcw8n\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-kube-api-access-zcw8n\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:43.851708 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:43.851673 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:44.190839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:44.190811 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg"] Nov 03 22:58:44.193180 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:58:44.193130 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ab07a9_138a_4e7d_83a8_fbe9ee4944d5.slice/crio-d81541ff7661a7ad5fe3952d98834ceecf3db71a2f1df836ae743305e120a12e WatchSource:0}: Error finding container d81541ff7661a7ad5fe3952d98834ceecf3db71a2f1df836ae743305e120a12e: Status 404 returned error can't find the container with id d81541ff7661a7ad5fe3952d98834ceecf3db71a2f1df836ae743305e120a12e Nov 03 22:58:44.196895 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:44.196859 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:58:44.197011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:44.196943 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:58:44.197011 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:44.196981 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:58:44.886329 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:44.886295 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" event={"ID":"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5","Type":"ContainerStarted","Data":"5cc74aaaffc275032c70d3f8a12beb8c2da6f7ed456adfdcdd63804552a04d02"} Nov 03 22:58:44.886754 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:44.886352 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" event={"ID":"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5","Type":"ContainerStarted","Data":"d81541ff7661a7ad5fe3952d98834ceecf3db71a2f1df836ae743305e120a12e"} Nov 03 22:58:44.967902 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:44.967835 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" podStartSLOduration=1.9678195569999999 podStartE2EDuration="1.967819557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:44.951489068 +0000 UTC m=+2746.380825060" watchObservedRunningTime="2026-11-03 22:58:44.967819557 +0000 UTC m=+2746.397155549" Nov 03 22:58:45.861817 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:45.861093 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:45.874839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:45.873908 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 22:58:45.874839 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:45.873973 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 22:58:45.881069 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:45.879845 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 22:58:46.858198 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:46.858145 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 22:58:46.858668 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:46.858223 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 22:58:47.854739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:47.854124 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 22:58:47.854739 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:47.854201 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 22:58:48.859992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:48.859956 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 22:58:48.860600 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:48.860567 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 22:58:49.853114 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:49.853075 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 22:58:49.853393 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:49.853364 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 22:58:50.857498 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:50.857464 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:50.858051 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:50.857961 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:58:50.858855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:58:50.858832 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 22:59:02.189070 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:59:02.189035 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:59:20.258928 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:20.258887 4625 generic.go:356] "Generic (PLEG): container finished" podID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerID="6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd" exitCode=0 Nov 03 22:59:20.259480 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:20.258960 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" event={"ID":"25766c62-3e88-493d-bfeb-a3e33b86e67c","Type":"ContainerDied","Data":"6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd"} Nov 03 22:59:20.259554 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:20.259514 4625 scope.go:122] "RemoveContainer" containerID="6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd" Nov 03 22:59:20.261841 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:20.261809 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:59:20.261943 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:20.261899 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:59:20.261992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:20.261943 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:59:21.281670 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:21.281630 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" event={"ID":"25766c62-3e88-493d-bfeb-a3e33b86e67c","Type":"ContainerStarted","Data":"62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e"} Nov 03 22:59:22.125646 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:22.125610 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:59:22.125881 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:22.125803 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 22:59:22.125964 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:22.125872 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 22:59:23.126195 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:23.126154 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 22:59:23.126590 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:23.126222 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 22:59:23.899988 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:23.899949 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w"] Nov 03 22:59:24.129607 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:24.129566 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 22:59:24.129992 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:24.129626 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 22:59:25.126025 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:25.125992 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 22:59:25.126239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:25.126044 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 22:59:26.125929 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:26.125895 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 03 22:59:26.126355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:26.125957 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 03 22:59:27.130585 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:27.130552 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:59:27.131128 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:27.131054 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:59:27.131912 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:27.131886 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 22:59:32.187141 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:59:32.187106 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 22:59:39.117676 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.117608 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" podUID="6f5692af-1f8c-41db-bc03-7c53104de383" containerName="main" containerID="cri-o://8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7" gracePeriod=45 Nov 03 22:59:39.426931 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.426907 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:59:39.443635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.443599 4625 generic.go:356] "Generic (PLEG): container finished" podID="6f5692af-1f8c-41db-bc03-7c53104de383" containerID="8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7" exitCode=0 Nov 03 22:59:39.443855 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.443829 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" event={"ID":"6f5692af-1f8c-41db-bc03-7c53104de383","Type":"ContainerDied","Data":"8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7"} Nov 03 22:59:39.443937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.443868 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" event={"ID":"6f5692af-1f8c-41db-bc03-7c53104de383","Type":"ContainerDied","Data":"97c712d9f07adbb478ea866389cacc84097c8136425ebfa671db5a24bbbf4e99"} Nov 03 22:59:39.443937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.443889 4625 scope.go:122] "RemoveContainer" containerID="8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7" Nov 03 22:59:39.444049 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.444034 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w" Nov 03 22:59:39.458189 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.458164 4625 scope.go:122] "RemoveContainer" containerID="8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7" Nov 03 22:59:39.458565 ip-10-0-128-81 kubenswrapper[4625]: E1103 22:59:39.458535 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7\": container with ID starting with 8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7 not found: ID does not exist" containerID="8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7" Nov 03 22:59:39.458645 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.458577 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7"} err="failed to get container status \"8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7\": rpc error: code = NotFound desc = could not find container \"8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7\": container with ID starting with 8f4a83a3465a835ef2bec2c46eead321116e0654614c3bcb8a5d48eae03ff5e7 not found: ID does not exist" Nov 03 22:59:39.509951 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.509915 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-home\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-home\") pod \"6f5692af-1f8c-41db-bc03-7c53104de383\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " Nov 03 22:59:39.510130 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.509967 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5692af-1f8c-41db-bc03-7c53104de383-kube-api-access-tl79q\" (UniqueName: \"kubernetes.io/projected/6f5692af-1f8c-41db-bc03-7c53104de383-kube-api-access-tl79q\") pod \"6f5692af-1f8c-41db-bc03-7c53104de383\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " Nov 03 22:59:39.510182 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.510149 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-model-cache\") pod \"6f5692af-1f8c-41db-bc03-7c53104de383\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " Nov 03 22:59:39.510242 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.510202 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-home" pod "6f5692af-1f8c-41db-bc03-7c53104de383" (UID: "6f5692af-1f8c-41db-bc03-7c53104de383"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:39.510302 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.510274 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-tmp-dir\") pod \"6f5692af-1f8c-41db-bc03-7c53104de383\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " Nov 03 22:59:39.510356 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.510302 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-dshm\") pod \"6f5692af-1f8c-41db-bc03-7c53104de383\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " Nov 03 22:59:39.510412 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.510352 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5692af-1f8c-41db-bc03-7c53104de383-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5692af-1f8c-41db-bc03-7c53104de383-tls-certs\") pod \"6f5692af-1f8c-41db-bc03-7c53104de383\" (UID: \"6f5692af-1f8c-41db-bc03-7c53104de383\") " Nov 03 22:59:39.510469 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.510447 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-model-cache" pod "6f5692af-1f8c-41db-bc03-7c53104de383" (UID: "6f5692af-1f8c-41db-bc03-7c53104de383"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:39.510579 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.510549 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-tmp-dir" pod "6f5692af-1f8c-41db-bc03-7c53104de383" (UID: "6f5692af-1f8c-41db-bc03-7c53104de383"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:39.511065 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.511022 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:39.511065 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.511046 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:39.511259 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.511083 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:39.517798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.517443 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5692af-1f8c-41db-bc03-7c53104de383-tls-certs" pod "6f5692af-1f8c-41db-bc03-7c53104de383" (UID: "6f5692af-1f8c-41db-bc03-7c53104de383"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:39.517798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.517461 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-dshm" pod "6f5692af-1f8c-41db-bc03-7c53104de383" (UID: "6f5692af-1f8c-41db-bc03-7c53104de383"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:39.517798 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.517472 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5692af-1f8c-41db-bc03-7c53104de383-kube-api-access-tl79q" pod "6f5692af-1f8c-41db-bc03-7c53104de383" (UID: "6f5692af-1f8c-41db-bc03-7c53104de383"). InnerVolumeSpecName "kube-api-access-tl79q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:39.611994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.611955 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5692af-1f8c-41db-bc03-7c53104de383-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:39.611994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.611984 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5692af-1f8c-41db-bc03-7c53104de383-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:39.611994 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.611995 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl79q\" (UniqueName: \"kubernetes.io/projected/6f5692af-1f8c-41db-bc03-7c53104de383-kube-api-access-tl79q\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:39.793209 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.793175 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w"] Nov 03 22:59:39.796973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:39.796949 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/sub-test-model-2a2eca-kserve-788cf5788f-6nx2w"] Nov 03 22:59:40.728364 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:40.728331 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5692af-1f8c-41db-bc03-7c53104de383" path="/var/lib/kubelet/pods/6f5692af-1f8c-41db-bc03-7c53104de383/volumes" Nov 03 22:59:43.497519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:43.497473 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh"] Nov 03 22:59:46.175197 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.175161 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk"] Nov 03 22:59:46.176575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.176543 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6f5692af-1f8c-41db-bc03-7c53104de383" containerName="main" Nov 03 22:59:46.176575 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.176570 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5692af-1f8c-41db-bc03-7c53104de383" containerName="main" Nov 03 22:59:46.176849 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.176807 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5692af-1f8c-41db-bc03-7c53104de383" containerName="main" Nov 03 22:59:46.181281 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.181255 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.224184 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.224154 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk"] Nov 03 22:59:46.229179 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.229143 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4a01f9-openshift-default-dockercfg-vznxz\"" Nov 03 22:59:46.253656 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.253617 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-podinfo\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.254046 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.254020 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnq5n\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-kube-api-access-pnq5n\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.254263 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.254241 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-socket\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.254483 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.254465 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-certs\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.254699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.254680 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-credential-socket\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.254896 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.254878 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istiod-ca-cert\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.255109 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.255092 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-data\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.255353 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.255329 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-token\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.255452 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.255381 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-envoy\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.372707 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.372662 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-token\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.373031 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.373005 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-envoy\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.373223 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.373207 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-podinfo\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.373378 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.373363 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnq5n\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-kube-api-access-pnq5n\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.373503 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.373489 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-socket\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.373659 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.373643 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-certs\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.373831 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.373814 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-credential-socket\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.373985 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.373967 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istiod-ca-cert\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.374149 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.374134 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-data\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.391738 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.374789 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-data\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.393041 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.393010 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-socket\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.393229 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.393020 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-credential-socket\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.393355 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.393244 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-certs\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.434786 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.394289 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istiod-ca-cert\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.435019 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.409390 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-envoy\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.435140 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.418881 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-podinfo\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.435239 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.432254 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-token\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.459742 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.459100 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnq5n\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-kube-api-access-pnq5n\") pod \"e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.559704 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.559663 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:46.990284 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.990245 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk"] Nov 03 22:59:46.991050 ip-10-0-128-81 kubenswrapper[4625]: W1103 22:59:46.991025 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a608e71_3ef4_4f7e_9e0c_e48c8e935037.slice/crio-92244396bd25ec1f67f5c6b36f752c0496e4e4e685166337d44af743269d3c91 WatchSource:0}: Error finding container 92244396bd25ec1f67f5c6b36f752c0496e4e4e685166337d44af743269d3c91: Status 404 returned error can't find the container with id 92244396bd25ec1f67f5c6b36f752c0496e4e4e685166337d44af743269d3c91 Nov 03 22:59:46.993931 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.993899 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:59:46.994042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.993964 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:59:46.994042 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:46.993995 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 22:59:47.594332 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:47.594295 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" event={"ID":"7a608e71-3ef4-4f7e-9e0c-e48c8e935037","Type":"ContainerStarted","Data":"31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e"} Nov 03 22:59:47.594763 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:47.594352 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" event={"ID":"7a608e71-3ef4-4f7e-9e0c-e48c8e935037","Type":"ContainerStarted","Data":"92244396bd25ec1f67f5c6b36f752c0496e4e4e685166337d44af743269d3c91"} Nov 03 22:59:48.560079 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:48.560040 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:48.567499 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:48.567472 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:48.601630 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:48.601599 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:48.602386 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:48.602368 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 22:59:48.625919 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:48.625871 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podStartSLOduration=2.625857274 podStartE2EDuration="2.625857274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:47.659564103 +0000 UTC m=+2809.088900095" watchObservedRunningTime="2026-11-03 22:59:48.625857274 +0000 UTC m=+2810.055193265" Nov 03 22:59:58.098525 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:58.098483 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9"] Nov 03 22:59:58.523252 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:58.523144 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" podUID="9f244f35-64d5-4c80-9f9b-41c558aae6a9" containerName="main" containerID="cri-o://a1f5b2b71d61ece52822d9f3f1c16ebf03b23504741dff8bd30fc35dda193ab4" gracePeriod=45 Nov 03 22:59:58.713096 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:58.712978 4625 generic.go:356] "Generic (PLEG): container finished" podID="9f244f35-64d5-4c80-9f9b-41c558aae6a9" containerID="a1f5b2b71d61ece52822d9f3f1c16ebf03b23504741dff8bd30fc35dda193ab4" exitCode=0 Nov 03 22:59:58.713326 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:58.713299 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" event={"ID":"9f244f35-64d5-4c80-9f9b-41c558aae6a9","Type":"ContainerDied","Data":"a1f5b2b71d61ece52822d9f3f1c16ebf03b23504741dff8bd30fc35dda193ab4"} Nov 03 22:59:58.878539 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:58.878514 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:59:59.030519 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030484 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-model-cache\") pod \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " Nov 03 22:59:59.030699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030582 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-home\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-home\") pod \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " Nov 03 22:59:59.030699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030642 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f244f35-64d5-4c80-9f9b-41c558aae6a9-kube-api-access-rwxhb\" (UniqueName: \"kubernetes.io/projected/9f244f35-64d5-4c80-9f9b-41c558aae6a9-kube-api-access-rwxhb\") pod \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " Nov 03 22:59:59.030699 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030676 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-dshm\") pod \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " Nov 03 22:59:59.030937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030779 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-model-cache" pod "9f244f35-64d5-4c80-9f9b-41c558aae6a9" (UID: "9f244f35-64d5-4c80-9f9b-41c558aae6a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:59.030937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030859 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-home" pod "9f244f35-64d5-4c80-9f9b-41c558aae6a9" (UID: "9f244f35-64d5-4c80-9f9b-41c558aae6a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:59.030937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030895 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tmp-dir\") pod \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " Nov 03 22:59:59.030937 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.030921 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tls-certs\") pod \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\" (UID: \"9f244f35-64d5-4c80-9f9b-41c558aae6a9\") " Nov 03 22:59:59.031142 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.031106 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tmp-dir" pod "9f244f35-64d5-4c80-9f9b-41c558aae6a9" (UID: "9f244f35-64d5-4c80-9f9b-41c558aae6a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:59.031491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.031471 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:59.031491 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.031490 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:59.031690 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.031499 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:59.032932 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.032905 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-dshm" pod "9f244f35-64d5-4c80-9f9b-41c558aae6a9" (UID: "9f244f35-64d5-4c80-9f9b-41c558aae6a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:59.033059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.032942 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f244f35-64d5-4c80-9f9b-41c558aae6a9-kube-api-access-rwxhb" pod "9f244f35-64d5-4c80-9f9b-41c558aae6a9" (UID: "9f244f35-64d5-4c80-9f9b-41c558aae6a9"). InnerVolumeSpecName "kube-api-access-rwxhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:59.033059 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.033041 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tls-certs" pod "9f244f35-64d5-4c80-9f9b-41c558aae6a9" (UID: "9f244f35-64d5-4c80-9f9b-41c558aae6a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:59.132581 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.132495 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f244f35-64d5-4c80-9f9b-41c558aae6a9-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:59.132581 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.132524 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwxhb\" (UniqueName: \"kubernetes.io/projected/9f244f35-64d5-4c80-9f9b-41c558aae6a9-kube-api-access-rwxhb\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:59.132581 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.132534 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f244f35-64d5-4c80-9f9b-41c558aae6a9-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 22:59:59.201394 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.201276 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l"] Nov 03 22:59:59.202247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.202225 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9f244f35-64d5-4c80-9f9b-41c558aae6a9" containerName="main" Nov 03 22:59:59.202247 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.202246 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f244f35-64d5-4c80-9f9b-41c558aae6a9" containerName="main" Nov 03 22:59:59.202406 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.202356 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f244f35-64d5-4c80-9f9b-41c558aae6a9" containerName="main" Nov 03 22:59:59.204973 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.204948 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.227078 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.227047 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a4b994-openshift-default-dockercfg-n7lmw\"" Nov 03 22:59:59.242266 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.242238 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l"] Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351050 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-socket\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351120 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvzp\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-kube-api-access-zhvzp\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351155 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-credential-socket\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351185 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-istio-token\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351206 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c506e01f-7155-4b48-8f64-50cbaef69332-istiod-ca-cert\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351242 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-envoy\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351304 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-data\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351411 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-certs\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.376736 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.351495 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c506e01f-7155-4b48-8f64-50cbaef69332-istio-podinfo\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.455940 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c506e01f-7155-4b48-8f64-50cbaef69332-istiod-ca-cert\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456005 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-envoy\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456088 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-data\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456227 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-certs\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456329 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c506e01f-7155-4b48-8f64-50cbaef69332-istio-podinfo\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456419 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-socket\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456475 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvzp\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-kube-api-access-zhvzp\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456512 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-credential-socket\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.456991 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.456547 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-istio-token\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.501379 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.492207 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c506e01f-7155-4b48-8f64-50cbaef69332-istiod-ca-cert\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.504768 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.502574 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-data\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.504968 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.504499 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-certs\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.505308 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.505289 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-credential-socket\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.505652 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.505635 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-socket\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.508711 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.508667 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-envoy\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.514055 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.509011 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c506e01f-7155-4b48-8f64-50cbaef69332-istio-podinfo\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.517619 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.517066 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-istio-token\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.538403 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.538375 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvzp\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-kube-api-access-zhvzp\") pod \"e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 22:59:59.722185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.722099 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" Nov 03 22:59:59.722185 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.722148 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh" event={"ID":"9f244f35-64d5-4c80-9f9b-41c558aae6a9","Type":"ContainerDied","Data":"425653f384798c0059d2026d997ea8812d49c48de3418a79953101e6b768e443"} Nov 03 22:59:59.722420 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.722191 4625 scope.go:122] "RemoveContainer" containerID="a1f5b2b71d61ece52822d9f3f1c16ebf03b23504741dff8bd30fc35dda193ab4" Nov 03 22:59:59.800241 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.800204 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh"] Nov 03 22:59:59.804635 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.804604 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/auth-test-model-23d281-kserve-78dcbd5765-f2fxh"] Nov 03 22:59:59.821584 ip-10-0-128-81 kubenswrapper[4625]: I1103 22:59:59.821559 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:00:00.204902 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.204686 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l"] Nov 03 23:00:00.217079 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:00:00.217043 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc506e01f_7155_4b48_8f64_50cbaef69332.slice/crio-eb64fe78444f810ac940cd3fd74224e2893056f25c64963841c02e73ce21569e WatchSource:0}: Error finding container eb64fe78444f810ac940cd3fd74224e2893056f25c64963841c02e73ce21569e: Status 404 returned error can't find the container with id eb64fe78444f810ac940cd3fd74224e2893056f25c64963841c02e73ce21569e Nov 03 23:00:00.220886 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.220841 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:00:00.221066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.220938 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:00:00.221066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.220981 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:00:00.395488 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.394396 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk"] Nov 03 23:00:00.405865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.405832 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk"] Nov 03 23:00:00.406044 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.405975 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.434256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.434227 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:00:00.517693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.512379 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f467c-a8ef-405c-bb60-19729c69492c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.517693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.512596 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqj8k\" (UniqueName: \"kubernetes.io/projected/9c9f467c-a8ef-405c-bb60-19729c69492c-kube-api-access-dqj8k\") pod \"maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.613812 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.613780 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f467c-a8ef-405c-bb60-19729c69492c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.614029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.613873 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqj8k\" (UniqueName: \"kubernetes.io/projected/9c9f467c-a8ef-405c-bb60-19729c69492c-kube-api-access-dqj8k\") pod \"maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.614866 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.614839 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f467c-a8ef-405c-bb60-19729c69492c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.640698 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.640668 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqj8k\" (UniqueName: \"kubernetes.io/projected/9c9f467c-a8ef-405c-bb60-19729c69492c-kube-api-access-dqj8k\") pod \"maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.744739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.741314 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:00.769290 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.769032 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f244f35-64d5-4c80-9f9b-41c558aae6a9" path="/var/lib/kubelet/pods/9f244f35-64d5-4c80-9f9b-41c558aae6a9/volumes" Nov 03 23:00:00.770378 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.770259 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" event={"ID":"c506e01f-7155-4b48-8f64-50cbaef69332","Type":"ContainerStarted","Data":"a595c912d64e188415412768e14d46ed825c590ae9a0707b46db45b2684eb935"} Nov 03 23:00:00.770378 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.770314 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" event={"ID":"c506e01f-7155-4b48-8f64-50cbaef69332","Type":"ContainerStarted","Data":"eb64fe78444f810ac940cd3fd74224e2893056f25c64963841c02e73ce21569e"} Nov 03 23:00:00.823493 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.823463 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:00:00.825306 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:00.825255 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podStartSLOduration=1.825239644 podStartE2EDuration="1.825239644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:00.817664854 +0000 UTC m=+2822.247000846" watchObservedRunningTime="2026-11-03 23:00:00.825239644 +0000 UTC m=+2822.254575641" Nov 03 23:00:01.122760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:01.122687 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk"] Nov 03 23:00:01.126419 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:00:01.126381 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9f467c_a8ef_405c_bb60_19729c69492c.slice/crio-d832b44a1cf82e1e2a227ddd1986703e0d9d110c7053238b47d4714ed65e8ce9 WatchSource:0}: Error finding container d832b44a1cf82e1e2a227ddd1986703e0d9d110c7053238b47d4714ed65e8ce9: Status 404 returned error can't find the container with id d832b44a1cf82e1e2a227ddd1986703e0d9d110c7053238b47d4714ed65e8ce9 Nov 03 23:00:01.781335 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:01.781233 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" event={"ID":"9c9f467c-a8ef-405c-bb60-19729c69492c","Type":"ContainerStarted","Data":"3d0c19f571d3aaef8dd3e5d78fc4c15c3d51398c66a1274f6b9cd2913b23d43b"} Nov 03 23:00:01.781335 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:01.781326 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" event={"ID":"9c9f467c-a8ef-405c-bb60-19729c69492c","Type":"ContainerStarted","Data":"d832b44a1cf82e1e2a227ddd1986703e0d9d110c7053238b47d4714ed65e8ce9"} Nov 03 23:00:01.821040 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:01.820986 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" podStartSLOduration=1.820970906 podStartE2EDuration="1.820970906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:01.820400037 +0000 UTC m=+2823.249736028" watchObservedRunningTime="2026-11-03 23:00:01.820970906 +0000 UTC m=+2823.250306898" Nov 03 23:00:01.829761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:01.829733 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:00:02.191403 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:02.191360 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:00:02.803737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:02.801844 4625 generic.go:356] "Generic (PLEG): container finished" podID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerID="3d0c19f571d3aaef8dd3e5d78fc4c15c3d51398c66a1274f6b9cd2913b23d43b" exitCode=60 Nov 03 23:00:02.804868 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:02.804538 4625 scope.go:122] "RemoveContainer" containerID="3d0c19f571d3aaef8dd3e5d78fc4c15c3d51398c66a1274f6b9cd2913b23d43b" Nov 03 23:00:02.805026 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:02.804921 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" event={"ID":"9c9f467c-a8ef-405c-bb60-19729c69492c","Type":"ContainerDied","Data":"3d0c19f571d3aaef8dd3e5d78fc4c15c3d51398c66a1274f6b9cd2913b23d43b"} Nov 03 23:00:02.805026 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:02.804991 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:00:02.827020 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:02.826995 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:00:03.814779 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:03.814742 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" event={"ID":"9c9f467c-a8ef-405c-bb60-19729c69492c","Type":"ContainerStarted","Data":"b33cb3fde06d7e05aceb5d0f109f6361e1698a6014776d051a1b89464dfc2142"} Nov 03 23:00:04.824812 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:04.824780 4625 generic.go:356] "Generic (PLEG): container finished" podID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerID="b33cb3fde06d7e05aceb5d0f109f6361e1698a6014776d051a1b89464dfc2142" exitCode=60 Nov 03 23:00:04.825272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:04.824820 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" event={"ID":"9c9f467c-a8ef-405c-bb60-19729c69492c","Type":"ContainerDied","Data":"b33cb3fde06d7e05aceb5d0f109f6361e1698a6014776d051a1b89464dfc2142"} Nov 03 23:00:04.825272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:04.824888 4625 scope.go:122] "RemoveContainer" containerID="3d0c19f571d3aaef8dd3e5d78fc4c15c3d51398c66a1274f6b9cd2913b23d43b" Nov 03 23:00:04.825272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:04.825147 4625 scope.go:122] "RemoveContainer" containerID="b33cb3fde06d7e05aceb5d0f109f6361e1698a6014776d051a1b89464dfc2142" Nov 03 23:00:04.825530 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:04.825504 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk_odh-ai-gateway-infra(9c9f467c-a8ef-405c-bb60-19729c69492c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" Nov 03 23:00:05.834602 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:05.834577 4625 scope.go:122] "RemoveContainer" containerID="b33cb3fde06d7e05aceb5d0f109f6361e1698a6014776d051a1b89464dfc2142" Nov 03 23:00:05.835071 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:05.835032 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk_odh-ai-gateway-infra(9c9f467c-a8ef-405c-bb60-19729c69492c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" Nov 03 23:00:13.431048 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.430967 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" podUID="2baab7fc-28dd-4df9-a6e1-060b85470d72" containerName="main" containerID="cri-o://a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7" gracePeriod=45 Nov 03 23:00:13.734388 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.734353 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 23:00:13.863621 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.863587 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-tmp-dir\") pod \"2baab7fc-28dd-4df9-a6e1-060b85470d72\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " Nov 03 23:00:13.863870 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.863733 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baab7fc-28dd-4df9-a6e1-060b85470d72-kube-api-access-479f9\" (UniqueName: \"kubernetes.io/projected/2baab7fc-28dd-4df9-a6e1-060b85470d72-kube-api-access-479f9\") pod \"2baab7fc-28dd-4df9-a6e1-060b85470d72\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " Nov 03 23:00:13.863870 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.863806 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-model-cache\") pod \"2baab7fc-28dd-4df9-a6e1-060b85470d72\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " Nov 03 23:00:13.864003 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.863942 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2baab7fc-28dd-4df9-a6e1-060b85470d72-tls-certs\" (UniqueName: \"kubernetes.io/secret/2baab7fc-28dd-4df9-a6e1-060b85470d72-tls-certs\") pod \"2baab7fc-28dd-4df9-a6e1-060b85470d72\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " Nov 03 23:00:13.864003 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.863961 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-tmp-dir" pod "2baab7fc-28dd-4df9-a6e1-060b85470d72" (UID: "2baab7fc-28dd-4df9-a6e1-060b85470d72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:13.864114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.864054 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-home\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-home\") pod \"2baab7fc-28dd-4df9-a6e1-060b85470d72\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " Nov 03 23:00:13.864114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.864093 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-dshm\") pod \"2baab7fc-28dd-4df9-a6e1-060b85470d72\" (UID: \"2baab7fc-28dd-4df9-a6e1-060b85470d72\") " Nov 03 23:00:13.864114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.864102 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-model-cache" pod "2baab7fc-28dd-4df9-a6e1-060b85470d72" (UID: "2baab7fc-28dd-4df9-a6e1-060b85470d72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:13.864321 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.864294 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-home" pod "2baab7fc-28dd-4df9-a6e1-060b85470d72" (UID: "2baab7fc-28dd-4df9-a6e1-060b85470d72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:13.864954 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.864862 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:13.864954 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.864889 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:13.864954 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.864903 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:13.866492 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.866469 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baab7fc-28dd-4df9-a6e1-060b85470d72-kube-api-access-479f9" pod "2baab7fc-28dd-4df9-a6e1-060b85470d72" (UID: "2baab7fc-28dd-4df9-a6e1-060b85470d72"). InnerVolumeSpecName "kube-api-access-479f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:13.866606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.866524 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2baab7fc-28dd-4df9-a6e1-060b85470d72-tls-certs" pod "2baab7fc-28dd-4df9-a6e1-060b85470d72" (UID: "2baab7fc-28dd-4df9-a6e1-060b85470d72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:13.866606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.866531 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-dshm" pod "2baab7fc-28dd-4df9-a6e1-060b85470d72" (UID: "2baab7fc-28dd-4df9-a6e1-060b85470d72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:13.966732 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.966625 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baab7fc-28dd-4df9-a6e1-060b85470d72-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:13.966732 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.966658 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baab7fc-28dd-4df9-a6e1-060b85470d72-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:13.966732 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:13.966670 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479f9\" (UniqueName: \"kubernetes.io/projected/2baab7fc-28dd-4df9-a6e1-060b85470d72-kube-api-access-479f9\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:14.015833 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.015798 4625 generic.go:356] "Generic (PLEG): container finished" podID="2baab7fc-28dd-4df9-a6e1-060b85470d72" containerID="a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7" exitCode=0 Nov 03 23:00:14.015998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.015852 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" Nov 03 23:00:14.015998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.015892 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" event={"ID":"2baab7fc-28dd-4df9-a6e1-060b85470d72","Type":"ContainerDied","Data":"a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7"} Nov 03 23:00:14.015998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.015954 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9" event={"ID":"2baab7fc-28dd-4df9-a6e1-060b85470d72","Type":"ContainerDied","Data":"9bbbfb4e193ed00fcfa5ca37d672e97d70c6b5be3fa072396b4866e5f9705943"} Nov 03 23:00:14.015998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.015979 4625 scope.go:122] "RemoveContainer" containerID="a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7" Nov 03 23:00:14.085028 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.085002 4625 scope.go:122] "RemoveContainer" containerID="a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7" Nov 03 23:00:14.085395 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:14.085377 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7\": container with ID starting with a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7 not found: ID does not exist" containerID="a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7" Nov 03 23:00:14.085470 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.085409 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7"} err="failed to get container status \"a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7\": rpc error: code = NotFound desc = could not find container \"a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7\": container with ID starting with a660774b19475fce6670d141e05a7501c87b6676cd0e65e25ede4ef12a7fb0e7 not found: ID does not exist" Nov 03 23:00:14.122015 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.121977 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9"] Nov 03 23:00:14.125852 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.125827 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-9b841d/auth-test-model-9b841d-kserve-6b8dd89d8f-s9tf9"] Nov 03 23:00:14.729006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:14.728976 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2baab7fc-28dd-4df9-a6e1-060b85470d72" path="/var/lib/kubelet/pods/2baab7fc-28dd-4df9-a6e1-060b85470d72/volumes" Nov 03 23:00:15.720832 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:15.720806 4625 scope.go:122] "RemoveContainer" containerID="b33cb3fde06d7e05aceb5d0f109f6361e1698a6014776d051a1b89464dfc2142" Nov 03 23:00:17.043391 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:17.043298 4625 generic.go:356] "Generic (PLEG): container finished" podID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerID="44c95cf19317e8790363ab3acfc688e18a21dedb770479082600b0b92e3f4558" exitCode=60 Nov 03 23:00:17.043391 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:17.043375 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" event={"ID":"9c9f467c-a8ef-405c-bb60-19729c69492c","Type":"ContainerDied","Data":"44c95cf19317e8790363ab3acfc688e18a21dedb770479082600b0b92e3f4558"} Nov 03 23:00:17.043946 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:17.043437 4625 scope.go:122] "RemoveContainer" containerID="b33cb3fde06d7e05aceb5d0f109f6361e1698a6014776d051a1b89464dfc2142" Nov 03 23:00:17.044013 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:17.043957 4625 scope.go:122] "RemoveContainer" containerID="44c95cf19317e8790363ab3acfc688e18a21dedb770479082600b0b92e3f4558" Nov 03 23:00:17.044284 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:17.044260 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk_odh-ai-gateway-infra(9c9f467c-a8ef-405c-bb60-19729c69492c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" Nov 03 23:00:18.095562 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.095527 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk"] Nov 03 23:00:18.338551 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.338520 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:18.363950 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.363873 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9f467c-a8ef-405c-bb60-19729c69492c-kube-api-access-dqj8k\" (UniqueName: \"kubernetes.io/projected/9c9f467c-a8ef-405c-bb60-19729c69492c-kube-api-access-dqj8k\") pod \"9c9f467c-a8ef-405c-bb60-19729c69492c\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " Nov 03 23:00:18.364115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.363984 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9f467c-a8ef-405c-bb60-19729c69492c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f467c-a8ef-405c-bb60-19729c69492c-maas-api-service-ca\") pod \"9c9f467c-a8ef-405c-bb60-19729c69492c\" (UID: \"9c9f467c-a8ef-405c-bb60-19729c69492c\") " Nov 03 23:00:18.364396 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.364368 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9f467c-a8ef-405c-bb60-19729c69492c-maas-api-service-ca" pod "9c9f467c-a8ef-405c-bb60-19729c69492c" (UID: "9c9f467c-a8ef-405c-bb60-19729c69492c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:18.364711 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.364694 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9f467c-a8ef-405c-bb60-19729c69492c-maas-api-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:18.366410 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.366384 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9f467c-a8ef-405c-bb60-19729c69492c-kube-api-access-dqj8k" pod "9c9f467c-a8ef-405c-bb60-19729c69492c" (UID: "9c9f467c-a8ef-405c-bb60-19729c69492c"). InnerVolumeSpecName "kube-api-access-dqj8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:18.466169 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:18.466135 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqj8k\" (UniqueName: \"kubernetes.io/projected/9c9f467c-a8ef-405c-bb60-19729c69492c-kube-api-access-dqj8k\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:19.077107 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:19.077023 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" Nov 03 23:00:19.077107 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:19.077044 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk" event={"ID":"9c9f467c-a8ef-405c-bb60-19729c69492c","Type":"ContainerDied","Data":"d832b44a1cf82e1e2a227ddd1986703e0d9d110c7053238b47d4714ed65e8ce9"} Nov 03 23:00:19.077315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:19.077114 4625 scope.go:122] "RemoveContainer" containerID="44c95cf19317e8790363ab3acfc688e18a21dedb770479082600b0b92e3f4558" Nov 03 23:00:19.134507 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:19.134471 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk"] Nov 03 23:00:19.148197 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:19.148166 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f48f46-29895780-n72kk"] Nov 03 23:00:20.728902 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:20.728868 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" path="/var/lib/kubelet/pods/9c9f467c-a8ef-405c-bb60-19729c69492c/volumes" Nov 03 23:00:29.431143 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.431104 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj"] Nov 03 23:00:29.431882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.431863 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2baab7fc-28dd-4df9-a6e1-060b85470d72" containerName="main" Nov 03 23:00:29.431945 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.431884 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2baab7fc-28dd-4df9-a6e1-060b85470d72" containerName="main" Nov 03 23:00:29.431945 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.431894 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:00:29.431945 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.431900 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:00:29.431945 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.431914 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:00:29.431945 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.431920 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:00:29.432111 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.432059 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2baab7fc-28dd-4df9-a6e1-060b85470d72" containerName="main" Nov 03 23:00:29.432111 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.432066 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:00:29.432111 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.432073 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:00:29.435862 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.435839 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.453342 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.453313 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-0847aa-openshift-default-dockercfg-dkbjf\"" Nov 03 23:00:29.469168 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.469115 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.469429 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.469410 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.469581 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.469564 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7vt\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-kube-api-access-2g7vt\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.469786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.469757 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.469968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.469942 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.470248 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.470217 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dabd04-7e6e-46cd-9331-ced0975c71ca-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.470511 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.470481 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.470683 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.470662 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.470800 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.470762 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.472246 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.472220 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj"] Nov 03 23:00:29.572142 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572096 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572337 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572182 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572337 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572221 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572489 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572422 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7vt\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-kube-api-access-2g7vt\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572489 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572476 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572603 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572514 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572603 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572566 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572704 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dabd04-7e6e-46cd-9331-ced0975c71ca-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572800 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572745 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572858 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572805 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572858 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572814 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.572967 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.572884 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.573156 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.573132 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.573524 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.573501 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dabd04-7e6e-46cd-9331-ced0975c71ca-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.574936 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.574912 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.575438 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.575397 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.595315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.595282 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.595495 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.595477 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7vt\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-kube-api-access-2g7vt\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:29.758787 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:29.758112 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:30.137413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:30.137380 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj"] Nov 03 23:00:30.138529 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:00:30.138498 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dabd04_7e6e_46cd_9331_ced0975c71ca.slice/crio-5dccd62076e0b51f88e838b49e94103f454218f58475af7edc2d09232d17a7d6 WatchSource:0}: Error finding container 5dccd62076e0b51f88e838b49e94103f454218f58475af7edc2d09232d17a7d6: Status 404 returned error can't find the container with id 5dccd62076e0b51f88e838b49e94103f454218f58475af7edc2d09232d17a7d6 Nov 03 23:00:30.142093 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:30.142058 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:00:30.142227 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:30.142124 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:00:30.142227 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:30.142151 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:00:30.173819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:30.173787 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" event={"ID":"89dabd04-7e6e-46cd-9331-ced0975c71ca","Type":"ContainerStarted","Data":"5dccd62076e0b51f88e838b49e94103f454218f58475af7edc2d09232d17a7d6"} Nov 03 23:00:31.184340 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:31.184298 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" event={"ID":"89dabd04-7e6e-46cd-9331-ced0975c71ca","Type":"ContainerStarted","Data":"ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c"} Nov 03 23:00:31.272091 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:31.272011 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podStartSLOduration=2.271989476 podStartE2EDuration="2.271989476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:31.253245357 +0000 UTC m=+2852.682581349" watchObservedRunningTime="2026-11-03 23:00:31.271989476 +0000 UTC m=+2852.701325468" Nov 03 23:00:31.760739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:31.759859 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:31.766760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:31.763711 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 23:00:31.766760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:31.763828 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 23:00:32.197547 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:32.197509 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:00:32.762736 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:32.762084 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 23:00:32.762736 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:32.762158 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 23:00:33.760534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:33.760502 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 23:00:33.761138 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:33.761109 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 23:00:34.764151 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:34.764115 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 23:00:34.764640 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:34.764607 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 23:00:35.769740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:35.769123 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 23:00:35.769740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:35.769188 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 23:00:36.760029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:36.759306 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 23:00:36.760029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:36.759372 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 23:00:37.764291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:37.764260 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:37.764846 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:37.764761 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:37.765596 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:37.765575 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:00:38.497592 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:38.497556 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d"] Nov 03 23:00:38.497852 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:38.497822 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" podUID="66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" containerName="maas-api" containerID="cri-o://99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab" gracePeriod=30 Nov 03 23:00:38.849243 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:38.849212 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 23:00:39.043743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.039230 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-kube-api-access-hqpv4\" (UniqueName: \"kubernetes.io/projected/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-kube-api-access-hqpv4\") pod \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " Nov 03 23:00:39.043743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.039331 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-openshift-service-ca\") pod \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " Nov 03 23:00:39.043743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.039566 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-maas-api-tls\") pod \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " Nov 03 23:00:39.043743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.039604 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-metrics-certs\") pod \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\" (UID: \"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba\") " Nov 03 23:00:39.049099 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.049034 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-openshift-service-ca" pod "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" (UID: "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:39.088291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.088254 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-metrics-certs" pod "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" (UID: "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:39.094928 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.094895 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-maas-api-tls" pod "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" (UID: "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:39.112461 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.112426 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-kube-api-access-hqpv4" pod "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" (UID: "66f1b9f9-0bd1-48fe-841e-8d51d17db9ba"). InnerVolumeSpecName "kube-api-access-hqpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:39.146064 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.146027 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.146360 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.146342 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.146471 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.146460 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.146577 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.146563 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqpv4\" (UniqueName: \"kubernetes.io/projected/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba-kube-api-access-hqpv4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.320536 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.320438 4625 generic.go:356] "Generic (PLEG): container finished" podID="66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" containerID="99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab" exitCode=0 Nov 03 23:00:39.322279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.321452 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" Nov 03 23:00:39.322279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.321984 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" event={"ID":"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba","Type":"ContainerDied","Data":"99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab"} Nov 03 23:00:39.322279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.322033 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d" event={"ID":"66f1b9f9-0bd1-48fe-841e-8d51d17db9ba","Type":"ContainerDied","Data":"8e51f6c81172fa369b241a0d27663072f24a116dc4086b50931a3add80bb7314"} Nov 03 23:00:39.322279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.322053 4625 scope.go:122] "RemoveContainer" containerID="99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab" Nov 03 23:00:39.348260 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.347995 4625 scope.go:122] "RemoveContainer" containerID="99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab" Nov 03 23:00:39.348583 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:39.348496 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab\": container with ID starting with 99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab not found: ID does not exist" containerID="99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab" Nov 03 23:00:39.348583 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.348534 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab"} err="failed to get container status \"99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab\": rpc error: code = NotFound desc = could not find container \"99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab\": container with ID starting with 99d7b3db86dab620df3a7f35127358ff8addb95b10935ce02ea8fe2292018fab not found: ID does not exist" Nov 03 23:00:39.433119 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.401068 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d"] Nov 03 23:00:39.433119 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:39.402812 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-68410c-c76ff7b88-5974d"] Nov 03 23:00:40.728981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:40.728948 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" path="/var/lib/kubelet/pods/66f1b9f9-0bd1-48fe-841e-8d51d17db9ba/volumes" Nov 03 23:00:46.939740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:46.934821 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4"] Nov 03 23:00:46.939740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:46.935109 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" containerID="cri-o://6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9" gracePeriod=30 Nov 03 23:00:52.270871 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.270837 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 23:00:52.344807 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.344776 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-data\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.345004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.344816 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-token\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-token\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.345004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.344928 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-socket\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.345004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.344973 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-envoy\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.345181 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.345050 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-certs\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.345538 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.345255 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-socket" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:52.345538 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.345331 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-data" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:52.345538 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.345502 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-certs" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:52.345883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.345866 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-podinfo\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346006 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-kube-api-access-5dzdr\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-kube-api-access-5dzdr\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346182 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-credential-socket\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346256 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b89cf6c-f171-4e05-a770-63e6d790eef8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b89cf6c-f171-4e05-a770-63e6d790eef8-istiod-ca-cert\") pod \"1b89cf6c-f171-4e05-a770-63e6d790eef8\" (UID: \"1b89cf6c-f171-4e05-a770-63e6d790eef8\") " Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346646 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-credential-socket" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346866 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b89cf6c-f171-4e05-a770-63e6d790eef8-istiod-ca-cert" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346942 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346957 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346967 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.346994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.346975 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.349113 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.349085 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-envoy" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:52.349540 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.349515 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-podinfo" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:52.349801 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.349777 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-token" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:52.350246 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.350219 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-kube-api-access-5dzdr" pod "1b89cf6c-f171-4e05-a770-63e6d790eef8" (UID: "1b89cf6c-f171-4e05-a770-63e6d790eef8"). InnerVolumeSpecName "kube-api-access-5dzdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:52.448452 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.448421 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b89cf6c-f171-4e05-a770-63e6d790eef8-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.448452 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.448448 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.448731 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.448462 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.448731 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.448473 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b89cf6c-f171-4e05-a770-63e6d790eef8-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.448731 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.448483 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dzdr\" (UniqueName: \"kubernetes.io/projected/1b89cf6c-f171-4e05-a770-63e6d790eef8-kube-api-access-5dzdr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:00:52.686267 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.686180 4625 generic.go:356] "Generic (PLEG): container finished" podID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerID="6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9" exitCode=0 Nov 03 23:00:52.686267 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.686246 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" Nov 03 23:00:52.686462 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.686265 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" event={"ID":"1b89cf6c-f171-4e05-a770-63e6d790eef8","Type":"ContainerDied","Data":"6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9"} Nov 03 23:00:52.686462 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.686329 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4" event={"ID":"1b89cf6c-f171-4e05-a770-63e6d790eef8","Type":"ContainerDied","Data":"134b1829df192c635b3d8e65f9db59bf80584c84fff758cc09af0f77a324702b"} Nov 03 23:00:52.686462 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.686353 4625 scope.go:122] "RemoveContainer" containerID="6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9" Nov 03 23:00:52.698700 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.698680 4625 scope.go:122] "RemoveContainer" containerID="8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc" Nov 03 23:00:52.708821 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.708803 4625 scope.go:122] "RemoveContainer" containerID="6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9" Nov 03 23:00:52.709165 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:52.709136 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9\": container with ID starting with 6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9 not found: ID does not exist" containerID="6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9" Nov 03 23:00:52.710692 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.709182 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9"} err="failed to get container status \"6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9\": rpc error: code = NotFound desc = could not find container \"6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9\": container with ID starting with 6a02091f7a2514a34b7dd7b400d07333078766b4d91ebec3fbf221813d583ac9 not found: ID does not exist" Nov 03 23:00:52.710856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.710707 4625 scope.go:122] "RemoveContainer" containerID="8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc" Nov 03 23:00:52.711245 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:00:52.711221 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc\": container with ID starting with 8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc not found: ID does not exist" containerID="8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc" Nov 03 23:00:52.711326 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.711252 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc"} err="failed to get container status \"8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc\": rpc error: code = NotFound desc = could not find container \"8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc\": container with ID starting with 8401e58a590c8d052b9036868e5b396d6a4235420770c892ea3a7d2d0e95accc not found: ID does not exist" Nov 03 23:00:52.739834 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.739804 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4"] Nov 03 23:00:52.752516 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:52.752490 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-68410c-openshift-default-7848d9f579-cqpd4"] Nov 03 23:00:54.730087 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:00:54.730055 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" path="/var/lib/kubelet/pods/1b89cf6c-f171-4e05-a770-63e6d790eef8/volumes" Nov 03 23:01:01.178054 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.177962 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc"] Nov 03 23:01:01.178588 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.178227 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" podUID="ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" containerName="maas-api" containerID="cri-o://bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1" gracePeriod=30 Nov 03 23:01:01.480580 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.480553 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 23:01:01.587981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.587949 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-kube-api-access-s9jlm\" (UniqueName: \"kubernetes.io/projected/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-kube-api-access-s9jlm\") pod \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " Nov 03 23:01:01.588164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.588071 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-openshift-service-ca\") pod \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " Nov 03 23:01:01.588244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.588230 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-maas-api-tls\") pod \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " Nov 03 23:01:01.588289 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.588272 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-metrics-certs\") pod \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\" (UID: \"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf\") " Nov 03 23:01:01.588478 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.588448 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-openshift-service-ca" pod "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" (UID: "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:01.588799 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.588771 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:01.590069 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.590046 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-kube-api-access-s9jlm" pod "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" (UID: "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf"). InnerVolumeSpecName "kube-api-access-s9jlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:01.590206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.590190 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-maas-api-tls" pod "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" (UID: "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:01.590258 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.590238 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-metrics-certs" pod "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" (UID: "ed89659f-e1b0-4c90-b1e8-35a8518fc4cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:01.693737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.692973 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:01.693737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.693009 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:01.693737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.693022 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jlm\" (UniqueName: \"kubernetes.io/projected/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf-kube-api-access-s9jlm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:01.898868 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.896022 4625 generic.go:356] "Generic (PLEG): container finished" podID="ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" containerID="bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1" exitCode=0 Nov 03 23:01:01.898868 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.896076 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" event={"ID":"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf","Type":"ContainerDied","Data":"bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1"} Nov 03 23:01:01.898868 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.896132 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" event={"ID":"ed89659f-e1b0-4c90-b1e8-35a8518fc4cf","Type":"ContainerDied","Data":"dc94c1882c22ee4898d455aac386f8d0382edef12430835ee4319703e764c137"} Nov 03 23:01:01.898868 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.896166 4625 scope.go:122] "RemoveContainer" containerID="bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1" Nov 03 23:01:01.898868 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.896360 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc" Nov 03 23:01:01.942929 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.942905 4625 scope.go:122] "RemoveContainer" containerID="bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1" Nov 03 23:01:01.944120 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:01:01.943660 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1\": container with ID starting with bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1 not found: ID does not exist" containerID="bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1" Nov 03 23:01:01.944302 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.944261 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1"} err="failed to get container status \"bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1\": rpc error: code = NotFound desc = could not find container \"bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1\": container with ID starting with bafc59c1b09b76b280b1fc2d9e2c8d4c5f0d694b30fbc422a4eab2361b9bdff1 not found: ID does not exist" Nov 03 23:01:01.979646 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.979611 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc"] Nov 03 23:01:01.987601 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:01.987573 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff80c0-6bbddfbff6-rhpdc"] Nov 03 23:01:02.186446 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:01:02.186361 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:01:02.729494 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:02.729462 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" path="/var/lib/kubelet/pods/ed89659f-e1b0-4c90-b1e8-35a8518fc4cf/volumes" Nov 03 23:01:08.830176 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:08.830141 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz"] Nov 03 23:01:09.870735 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:09.870688 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg"] Nov 03 23:01:09.871411 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:09.870989 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" containerID="cri-o://5cc74aaaffc275032c70d3f8a12beb8c2da6f7ed456adfdcdd63804552a04d02" gracePeriod=30 Nov 03 23:01:15.118326 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.118293 4625 generic.go:356] "Generic (PLEG): container finished" podID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerID="5cc74aaaffc275032c70d3f8a12beb8c2da6f7ed456adfdcdd63804552a04d02" exitCode=0 Nov 03 23:01:15.118801 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.118391 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" event={"ID":"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5","Type":"ContainerDied","Data":"5cc74aaaffc275032c70d3f8a12beb8c2da6f7ed456adfdcdd63804552a04d02"} Nov 03 23:01:15.216473 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.216443 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 23:01:15.280689 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.280656 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-credential-socket\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.280911 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.280706 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istiod-ca-cert\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.280911 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.280774 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-token\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-token\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.281050 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281003 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-credential-socket" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.281050 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281012 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-envoy\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.281170 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281057 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-certs\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.281170 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281109 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-socket\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.281170 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281137 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istiod-ca-cert" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:15.281170 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281145 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-data\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.281387 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281218 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-podinfo\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.281387 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281253 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-kube-api-access-zcw8n\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-kube-api-access-zcw8n\") pod \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\" (UID: \"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5\") " Nov 03 23:01:15.281634 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281348 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-data" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.281869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.281846 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-socket" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.282140 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.282118 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.282260 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.282143 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.282260 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.282158 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.282260 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.282170 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.283971 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.283079 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-certs" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.285166 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.285146 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-kube-api-access-zcw8n" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "kube-api-access-zcw8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:15.285281 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.285177 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-token" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:15.285281 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.285198 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-envoy" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.285527 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.285510 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-podinfo" pod "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" (UID: "64ab07a9-138a-4e7d-83a8-fbe9ee4944d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:15.316004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.315964 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9"] Nov 03 23:01:15.383540 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.383505 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.383540 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.383538 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.383540 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.383548 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.383807 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.383556 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.383807 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:15.383565 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcw8n\" (UniqueName: \"kubernetes.io/projected/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5-kube-api-access-zcw8n\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:16.174736 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:16.171385 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" event={"ID":"64ab07a9-138a-4e7d-83a8-fbe9ee4944d5","Type":"ContainerDied","Data":"d81541ff7661a7ad5fe3952d98834ceecf3db71a2f1df836ae743305e120a12e"} Nov 03 23:01:16.174736 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:16.171458 4625 scope.go:122] "RemoveContainer" containerID="5cc74aaaffc275032c70d3f8a12beb8c2da6f7ed456adfdcdd63804552a04d02" Nov 03 23:01:16.174736 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:16.171615 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg" Nov 03 23:01:16.266236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:16.266205 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg"] Nov 03 23:01:16.273950 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:16.273287 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-z2mwg"] Nov 03 23:01:16.769760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:16.766929 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" path="/var/lib/kubelet/pods/64ab07a9-138a-4e7d-83a8-fbe9ee4944d5/volumes" Nov 03 23:01:22.190742 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:22.190226 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5"] Nov 03 23:01:23.914615 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:23.914183 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" podUID="bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" containerName="main" containerID="cri-o://ad8b0b40a3339ff92d76919ce64bbf980a2da1b3ad0a967acfef231ca75e9e4f" gracePeriod=45 Nov 03 23:01:24.104254 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:01:24.104210 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdc38cc9_0fa8_4546_86e0_1c93f3872a2f.slice/crio-ad8b0b40a3339ff92d76919ce64bbf980a2da1b3ad0a967acfef231ca75e9e4f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 23:01:24.343688 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.343654 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" event={"ID":"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f","Type":"ContainerDied","Data":"ad8b0b40a3339ff92d76919ce64bbf980a2da1b3ad0a967acfef231ca75e9e4f"} Nov 03 23:01:24.343891 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.343615 4625 generic.go:356] "Generic (PLEG): container finished" podID="bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" containerID="ad8b0b40a3339ff92d76919ce64bbf980a2da1b3ad0a967acfef231ca75e9e4f" exitCode=0 Nov 03 23:01:24.377996 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.377959 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 23:01:24.440097 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.439582 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-dshm\") pod \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " Nov 03 23:01:24.440097 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.439705 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-model-cache\") pod \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " Nov 03 23:01:24.440097 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.439768 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-home\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-home\") pod \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " Nov 03 23:01:24.440097 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.439837 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-kube-api-access-w8ddw\" (UniqueName: \"kubernetes.io/projected/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-kube-api-access-w8ddw\") pod \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " Nov 03 23:01:24.440435 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.440341 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-model-cache" pod "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" (UID: "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:24.440673 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.440645 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tmp-dir\") pod \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " Nov 03 23:01:24.440819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.440794 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tls-certs\") pod \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\" (UID: \"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f\") " Nov 03 23:01:24.441493 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.441435 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:24.442206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.442109 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-home" pod "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" (UID: "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:24.442206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.442180 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tmp-dir" pod "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" (UID: "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:24.445688 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.445650 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tls-certs" pod "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" (UID: "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:24.461200 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.461171 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-kube-api-access-w8ddw" pod "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" (UID: "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f"). InnerVolumeSpecName "kube-api-access-w8ddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:24.470291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.470239 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-dshm" pod "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" (UID: "bdc38cc9-0fa8-4546-86e0-1c93f3872a2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:24.543935 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.543887 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:24.543935 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.543925 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8ddw\" (UniqueName: \"kubernetes.io/projected/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-kube-api-access-w8ddw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:24.543935 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.543940 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:24.544310 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.543953 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:24.544310 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:24.543965 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:25.356819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:25.355120 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" event={"ID":"bdc38cc9-0fa8-4546-86e0-1c93f3872a2f","Type":"ContainerDied","Data":"8ea74383df775b18aded6728b1d7701aab2a82171df392398ab98353ad3e92ec"} Nov 03 23:01:25.356819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:25.355170 4625 scope.go:122] "RemoveContainer" containerID="ad8b0b40a3339ff92d76919ce64bbf980a2da1b3ad0a967acfef231ca75e9e4f" Nov 03 23:01:25.356819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:25.355302 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz" Nov 03 23:01:25.409819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:25.409774 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz"] Nov 03 23:01:25.420271 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:25.420235 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0847aa/sub-test-model-0847aa-kserve-55469f7855-nq2fz"] Nov 03 23:01:26.730032 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:26.730002 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" path="/var/lib/kubelet/pods/bdc38cc9-0fa8-4546-86e0-1c93f3872a2f/volumes" Nov 03 23:01:27.191001 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.190927 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" podUID="b4d2e243-3b00-4807-8324-e14cb06f0d20" containerName="payload-pre-processing" containerID="cri-o://cb7707a1dcd6114da9ef7c8d1e095494aed9bf8effe0374ed5cfedb8b1cafc5f" gracePeriod=25 Nov 03 23:01:27.594161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.593924 4625 generic.go:356] "Generic (PLEG): container finished" podID="b4d2e243-3b00-4807-8324-e14cb06f0d20" containerID="cb7707a1dcd6114da9ef7c8d1e095494aed9bf8effe0374ed5cfedb8b1cafc5f" exitCode=0 Nov 03 23:01:27.594161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.594071 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" event={"ID":"b4d2e243-3b00-4807-8324-e14cb06f0d20","Type":"ContainerDied","Data":"cb7707a1dcd6114da9ef7c8d1e095494aed9bf8effe0374ed5cfedb8b1cafc5f"} Nov 03 23:01:27.634390 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.634358 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 23:01:27.737941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.737908 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d2e243-3b00-4807-8324-e14cb06f0d20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2e243-3b00-4807-8324-e14cb06f0d20-plugins-config-volume\") pod \"b4d2e243-3b00-4807-8324-e14cb06f0d20\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " Nov 03 23:01:27.739613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.738145 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d2e243-3b00-4807-8324-e14cb06f0d20-kube-api-access-s98jl\" (UniqueName: \"kubernetes.io/projected/b4d2e243-3b00-4807-8324-e14cb06f0d20-kube-api-access-s98jl\") pod \"b4d2e243-3b00-4807-8324-e14cb06f0d20\" (UID: \"b4d2e243-3b00-4807-8324-e14cb06f0d20\") " Nov 03 23:01:27.739613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.738337 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d2e243-3b00-4807-8324-e14cb06f0d20-plugins-config-volume" pod "b4d2e243-3b00-4807-8324-e14cb06f0d20" (UID: "b4d2e243-3b00-4807-8324-e14cb06f0d20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:27.739613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.738705 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d2e243-3b00-4807-8324-e14cb06f0d20-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:27.740613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.740580 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d2e243-3b00-4807-8324-e14cb06f0d20-kube-api-access-s98jl" pod "b4d2e243-3b00-4807-8324-e14cb06f0d20" (UID: "b4d2e243-3b00-4807-8324-e14cb06f0d20"). InnerVolumeSpecName "kube-api-access-s98jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:27.840711 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:27.840324 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s98jl\" (UniqueName: \"kubernetes.io/projected/b4d2e243-3b00-4807-8324-e14cb06f0d20-kube-api-access-s98jl\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.279476 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq"] Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280371 4625 cpu_manager.go:414] "containerMap: removing container" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280399 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280424 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280431 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280458 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280467 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280481 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b4d2e243-3b00-4807-8324-e14cb06f0d20" containerName="payload-pre-processing" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280487 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d2e243-3b00-4807-8324-e14cb06f0d20" containerName="payload-pre-processing" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280521 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" containerName="maas-api" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280527 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" containerName="maas-api" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280545 4625 cpu_manager.go:414] "containerMap: removing container" podUID="bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" containerName="main" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280552 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" containerName="main" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280570 4625 cpu_manager.go:414] "containerMap: removing container" podUID="66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" containerName="maas-api" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280576 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" containerName="maas-api" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280587 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280593 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280745 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed89659f-e1b0-4c90-b1e8-35a8518fc4cf" containerName="maas-api" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280759 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc38cc9-0fa8-4546-86e0-1c93f3872a2f" containerName="main" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280769 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f1b9f9-0bd1-48fe-841e-8d51d17db9ba" containerName="maas-api" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280779 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9f467c-a8ef-405c-bb60-19729c69492c" containerName="cleanup" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280788 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ab07a9-138a-4e7d-83a8-fbe9ee4944d5" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280797 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280810 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b89cf6c-f171-4e05-a770-63e6d790eef8" containerName="istio-proxy" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.280819 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d2e243-3b00-4807-8324-e14cb06f0d20" containerName="payload-pre-processing" Nov 03 23:01:28.283785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.283610 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.307497 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.307450 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4ad693-openshift-default-dockercfg-nvt7m\"" Nov 03 23:01:28.315641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.313508 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq"] Nov 03 23:01:28.348811 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.348378 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.349046 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.348841 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfp5\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-kube-api-access-vnfp5\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.349046 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.348969 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45159818-f7b5-45a8-b6b8-47873dfd02da-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.349046 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.349003 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.349314 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.349073 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.349555 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.349493 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.349555 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.349533 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.350196 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.349583 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.350196 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.349657 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.481954 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482014 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfp5\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-kube-api-access-vnfp5\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482062 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45159818-f7b5-45a8-b6b8-47873dfd02da-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482098 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482134 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482258 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482285 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482335 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482411 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.483817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.482952 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.486227 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.485839 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45159818-f7b5-45a8-b6b8-47873dfd02da-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.504761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.487751 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.504761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.488013 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.504761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.488090 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.611382 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.611341 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.641010 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.629569 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfp5\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-kube-api-access-vnfp5\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.641010 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.637160 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.641010 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.637630 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:28.676645 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.676611 4625 generic.go:356] "Generic (PLEG): container finished" podID="c506e01f-7155-4b48-8f64-50cbaef69332" containerID="a595c912d64e188415412768e14d46ed825c590ae9a0707b46db45b2684eb935" exitCode=0 Nov 03 23:01:28.677030 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.676905 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" event={"ID":"c506e01f-7155-4b48-8f64-50cbaef69332","Type":"ContainerDied","Data":"a595c912d64e188415412768e14d46ed825c590ae9a0707b46db45b2684eb935"} Nov 03 23:01:28.677693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.677677 4625 scope.go:122] "RemoveContainer" containerID="a595c912d64e188415412768e14d46ed825c590ae9a0707b46db45b2684eb935" Nov 03 23:01:28.681785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.680265 4625 generic.go:356] "Generic (PLEG): container finished" podID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerID="31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e" exitCode=0 Nov 03 23:01:28.682941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.680283 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" event={"ID":"7a608e71-3ef4-4f7e-9e0c-e48c8e935037","Type":"ContainerDied","Data":"31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e"} Nov 03 23:01:28.682941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.682385 4625 scope.go:122] "RemoveContainer" containerID="31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e" Nov 03 23:01:28.684027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.683989 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:28.684219 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.684198 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:28.684564 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.684541 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:28.688213 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.685853 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:28.688213 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.685925 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:28.688213 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.685974 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:28.712117 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.712082 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" Nov 03 23:01:28.721516 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.712083 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5" event={"ID":"b4d2e243-3b00-4807-8324-e14cb06f0d20","Type":"ContainerDied","Data":"1b164e64e1d174dee9530f0a8d2b7f3926c62034d58ddab44c99c890f080f88c"} Nov 03 23:01:28.721516 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.717927 4625 scope.go:122] "RemoveContainer" containerID="cb7707a1dcd6114da9ef7c8d1e095494aed9bf8effe0374ed5cfedb8b1cafc5f" Nov 03 23:01:28.858668 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.858635 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5"] Nov 03 23:01:28.887427 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.887394 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e19850-5898685ff-kd5l5"] Nov 03 23:01:28.900882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:28.900855 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:29.279056 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.278977 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq"] Nov 03 23:01:29.285676 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:01:29.285645 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45159818_f7b5_45a8_b6b8_47873dfd02da.slice/crio-079573f5228adbc439bd8e9b04f746e5baa5334de4fa2730801874634f240f6a WatchSource:0}: Error finding container 079573f5228adbc439bd8e9b04f746e5baa5334de4fa2730801874634f240f6a: Status 404 returned error can't find the container with id 079573f5228adbc439bd8e9b04f746e5baa5334de4fa2730801874634f240f6a Nov 03 23:01:29.289206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.289164 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:29.289301 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.289251 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:29.289340 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.289298 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:01:29.740745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.739859 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" event={"ID":"45159818-f7b5-45a8-b6b8-47873dfd02da","Type":"ContainerStarted","Data":"9eb2b8f41e7c5b5c2455b95bad80bfd9f33ad9826d35588a8cb2c35d407e58c2"} Nov 03 23:01:29.740745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.739925 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" event={"ID":"45159818-f7b5-45a8-b6b8-47873dfd02da","Type":"ContainerStarted","Data":"079573f5228adbc439bd8e9b04f746e5baa5334de4fa2730801874634f240f6a"} Nov 03 23:01:29.756740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.751237 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" event={"ID":"c506e01f-7155-4b48-8f64-50cbaef69332","Type":"ContainerStarted","Data":"34e7ed06c0529e1d5ae3a07264addb49ba2995187df83dce301d070157438eef"} Nov 03 23:01:29.774506 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.774458 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" event={"ID":"7a608e71-3ef4-4f7e-9e0c-e48c8e935037","Type":"ContainerStarted","Data":"74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be"} Nov 03 23:01:29.840882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.840850 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:01:29.848987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.847893 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 23:01:29.851108 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.851074 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 23:01:29.883252 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.883190 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podStartSLOduration=1.883168692 podStartE2EDuration="1.883168692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:29.829175464 +0000 UTC m=+2911.258511457" watchObservedRunningTime="2026-11-03 23:01:29.883168692 +0000 UTC m=+2911.312504684" Nov 03 23:01:29.904289 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:29.904096 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:30.355741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.353175 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" containerID="cri-o://39152919a228182e69358ae996a68dc08d303da6cdc9f803c169c55423daf23b" gracePeriod=45 Nov 03 23:01:30.418539 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.418507 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Nov 03 23:01:30.566740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.563199 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 23:01:30.566740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.563378 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 23:01:30.566740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.563427 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 23:01:30.748862 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.733268 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d2e243-3b00-4807-8324-e14cb06f0d20" path="/var/lib/kubelet/pods/b4d2e243-3b00-4807-8324-e14cb06f0d20/volumes" Nov 03 23:01:30.815196 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.814863 4625 generic.go:356] "Generic (PLEG): container finished" podID="ac55f107-7b16-4247-94b9-fd23281948ef" containerID="39152919a228182e69358ae996a68dc08d303da6cdc9f803c169c55423daf23b" exitCode=0 Nov 03 23:01:30.815196 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.814932 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" event={"ID":"ac55f107-7b16-4247-94b9-fd23281948ef","Type":"ContainerDied","Data":"39152919a228182e69358ae996a68dc08d303da6cdc9f803c169c55423daf23b"} Nov 03 23:01:30.822033 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.821941 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 23:01:30.822033 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.821987 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 23:01:30.922693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:30.922605 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:31.031685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.031306 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 23:01:31.121316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.121282 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-model-cache\") pod \"ac55f107-7b16-4247-94b9-fd23281948ef\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " Nov 03 23:01:31.121316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.121322 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-dshm\") pod \"ac55f107-7b16-4247-94b9-fd23281948ef\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " Nov 03 23:01:31.121524 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.121421 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac55f107-7b16-4247-94b9-fd23281948ef-kube-api-access-5w8q5\" (UniqueName: \"kubernetes.io/projected/ac55f107-7b16-4247-94b9-fd23281948ef-kube-api-access-5w8q5\") pod \"ac55f107-7b16-4247-94b9-fd23281948ef\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " Nov 03 23:01:31.121524 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.121473 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac55f107-7b16-4247-94b9-fd23281948ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f107-7b16-4247-94b9-fd23281948ef-tls-certs\") pod \"ac55f107-7b16-4247-94b9-fd23281948ef\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " Nov 03 23:01:31.121602 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.121542 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-tmp-dir\") pod \"ac55f107-7b16-4247-94b9-fd23281948ef\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " Nov 03 23:01:31.121602 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.121591 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-home\") pod \"ac55f107-7b16-4247-94b9-fd23281948ef\" (UID: \"ac55f107-7b16-4247-94b9-fd23281948ef\") " Nov 03 23:01:31.122306 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.122274 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-home" pod "ac55f107-7b16-4247-94b9-fd23281948ef" (UID: "ac55f107-7b16-4247-94b9-fd23281948ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:31.122306 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.122282 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-model-cache" pod "ac55f107-7b16-4247-94b9-fd23281948ef" (UID: "ac55f107-7b16-4247-94b9-fd23281948ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:31.122514 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.122405 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-tmp-dir" pod "ac55f107-7b16-4247-94b9-fd23281948ef" (UID: "ac55f107-7b16-4247-94b9-fd23281948ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:31.124547 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.124522 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac55f107-7b16-4247-94b9-fd23281948ef-kube-api-access-5w8q5" pod "ac55f107-7b16-4247-94b9-fd23281948ef" (UID: "ac55f107-7b16-4247-94b9-fd23281948ef"). InnerVolumeSpecName "kube-api-access-5w8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:31.124658 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.124620 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-dshm" pod "ac55f107-7b16-4247-94b9-fd23281948ef" (UID: "ac55f107-7b16-4247-94b9-fd23281948ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:31.124741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.124681 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac55f107-7b16-4247-94b9-fd23281948ef-tls-certs" pod "ac55f107-7b16-4247-94b9-fd23281948ef" (UID: "ac55f107-7b16-4247-94b9-fd23281948ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:31.225518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.225415 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:31.225518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.225457 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:31.225518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.225472 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8q5\" (UniqueName: \"kubernetes.io/projected/ac55f107-7b16-4247-94b9-fd23281948ef-kube-api-access-5w8q5\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:31.225518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.225488 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f107-7b16-4247-94b9-fd23281948ef-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:31.225518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.225503 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:31.225518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.225515 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f107-7b16-4247-94b9-fd23281948ef-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:01:31.561802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.561591 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 23:01:31.561802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.561650 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 23:01:31.822134 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.822050 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 23:01:31.822134 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.822104 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 23:01:31.823913 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.823894 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" Nov 03 23:01:31.824042 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.823933 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9" event={"ID":"ac55f107-7b16-4247-94b9-fd23281948ef","Type":"ContainerDied","Data":"967157f70402659a5ff197cbec68965d3e9ff17b4f6062feb894f7f4d6302db6"} Nov 03 23:01:31.824042 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.823981 4625 scope.go:122] "RemoveContainer" containerID="39152919a228182e69358ae996a68dc08d303da6cdc9f803c169c55423daf23b" Nov 03 23:01:31.824575 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.824558 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:31.827019 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.827001 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:01:31.900864 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.900823 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9"] Nov 03 23:01:31.905035 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:31.905008 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-2-w1-e19850-kserve-79948bc5b9-qb7b9"] Nov 03 23:01:32.231655 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:01:32.231609 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:01:32.568999 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:32.568816 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 23:01:32.568999 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:32.568874 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 23:01:32.729049 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:32.729016 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" path="/var/lib/kubelet/pods/ac55f107-7b16-4247-94b9-fd23281948ef/volumes" Nov 03 23:01:32.803627 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:32.803595 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:01:32.822513 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:32.822443 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 23:01:32.822513 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:32.822489 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 23:01:33.562739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:33.562063 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 23:01:33.562739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:33.562121 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 23:01:33.605347 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:33.605311 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 23:01:33.822288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:33.822207 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 23:01:33.822288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:33.822259 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 23:01:34.577245 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:34.577207 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 23:01:34.580319 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:34.580280 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 23:01:34.822342 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:34.822303 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 23:01:34.822533 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:34.822356 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 23:01:35.629738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:35.628202 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 23:01:35.656735 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:35.653341 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 23:01:36.028508 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:36.026969 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:01:36.041176 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:01:36.041130 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:02:02.502804 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:02:02.496018 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:02:02.507255 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.506447 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48"] Nov 03 23:02:02.509399 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.509375 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" Nov 03 23:02:02.509602 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.509574 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" Nov 03 23:02:02.510116 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.510099 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac55f107-7b16-4247-94b9-fd23281948ef" containerName="main" Nov 03 23:02:02.515880 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.515866 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48"] Nov 03 23:02:02.519530 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.516021 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:02.548058 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.548034 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4a01f9-dockercfg-pzb9q\"" Nov 03 23:02:02.548346 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.548331 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4a01f9\"" Nov 03 23:02:02.592265 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.592231 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971c6d57-a245-4ffc-8573-d70e30757ef5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:02.592435 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.592281 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dclz\" (UniqueName: \"kubernetes.io/projected/971c6d57-a245-4ffc-8573-d70e30757ef5-kube-api-access-2dclz\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:02.693126 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.693094 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971c6d57-a245-4ffc-8573-d70e30757ef5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:02.693305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.693141 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dclz\" (UniqueName: \"kubernetes.io/projected/971c6d57-a245-4ffc-8573-d70e30757ef5-kube-api-access-2dclz\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:02.693839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.693821 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971c6d57-a245-4ffc-8573-d70e30757ef5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:02.716518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.716490 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dclz\" (UniqueName: \"kubernetes.io/projected/971c6d57-a245-4ffc-8573-d70e30757ef5-kube-api-access-2dclz\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:02.834469 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:02.834390 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:03.149872 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:03.149844 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48"] Nov 03 23:02:03.150531 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:02:03.150507 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971c6d57_a245_4ffc_8573_d70e30757ef5.slice/crio-f06ad81a36a093c0eb1a6ffb9b84deec4b995cda69da46f1e9252207cb20bf6d WatchSource:0}: Error finding container f06ad81a36a093c0eb1a6ffb9b84deec4b995cda69da46f1e9252207cb20bf6d: Status 404 returned error can't find the container with id f06ad81a36a093c0eb1a6ffb9b84deec4b995cda69da46f1e9252207cb20bf6d Nov 03 23:02:03.152701 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:03.152683 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:02:03.517570 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:03.517484 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" event={"ID":"971c6d57-a245-4ffc-8573-d70e30757ef5","Type":"ContainerStarted","Data":"f06ad81a36a093c0eb1a6ffb9b84deec4b995cda69da46f1e9252207cb20bf6d"} Nov 03 23:02:05.538003 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:05.537905 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" event={"ID":"971c6d57-a245-4ffc-8573-d70e30757ef5","Type":"ContainerStarted","Data":"d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251"} Nov 03 23:02:05.538595 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:05.538574 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:05.586062 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:05.586011 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" podStartSLOduration=1.6429043829999999 podStartE2EDuration="3.585995669s" totalImagesPullingTime="1.943091286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:05.582696968 +0000 UTC m=+2947.012032962" watchObservedRunningTime="2026-11-03 23:02:05.585995669 +0000 UTC m=+2947.015331689" Nov 03 23:02:12.568897 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.568849 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb"] Nov 03 23:02:12.573933 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.573909 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.595342 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.595312 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb"] Nov 03 23:02:12.597132 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.597106 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4ad693-serving-cert\"" Nov 03 23:02:12.665432 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.665395 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp7d\" (UniqueName: \"kubernetes.io/projected/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-kube-api-access-bjp7d\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.665627 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.665450 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.665708 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.665663 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.665808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.665793 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.767351 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.767319 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp7d\" (UniqueName: \"kubernetes.io/projected/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-kube-api-access-bjp7d\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.767559 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.767360 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.767559 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.767422 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.767559 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.767483 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.768102 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.768067 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.770110 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.770086 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.770226 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.770104 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.796946 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.796922 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp7d\" (UniqueName: \"kubernetes.io/projected/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-kube-api-access-bjp7d\") pod \"maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:12.889549 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:12.889512 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:13.205256 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:02:13.205210 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1eacc9_48d4_437b_9421_b451ed4f7ebd.slice/crio-5d9a1206c9e488f8e18680c332f08d2cc036a838b411b8594a4f397565d2f8f0 WatchSource:0}: Error finding container 5d9a1206c9e488f8e18680c332f08d2cc036a838b411b8594a4f397565d2f8f0: Status 404 returned error can't find the container with id 5d9a1206c9e488f8e18680c332f08d2cc036a838b411b8594a4f397565d2f8f0 Nov 03 23:02:13.205862 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:13.205801 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb"] Nov 03 23:02:13.622236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:13.622205 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" event={"ID":"5f1eacc9-48d4-437b-9421-b451ed4f7ebd","Type":"ContainerStarted","Data":"fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194"} Nov 03 23:02:13.622633 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:13.622258 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" event={"ID":"5f1eacc9-48d4-437b-9421-b451ed4f7ebd","Type":"ContainerStarted","Data":"5d9a1206c9e488f8e18680c332f08d2cc036a838b411b8594a4f397565d2f8f0"} Nov 03 23:02:13.622633 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:13.622402 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:13.667269 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:13.667220 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" podStartSLOduration=1.343242265 podStartE2EDuration="1.667206363s" totalImagesPullingTime="323.964098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:13.664389661 +0000 UTC m=+2955.093725654" watchObservedRunningTime="2026-11-03 23:02:13.667206363 +0000 UTC m=+2955.096542353" Nov 03 23:02:16.304205 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.304174 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w"] Nov 03 23:02:16.307632 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.307600 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.332345 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.332315 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w"] Nov 03 23:02:16.338520 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.338492 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4b994-dockercfg-7q8pr\"" Nov 03 23:02:16.338678 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.338578 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4b994\"" Nov 03 23:02:16.393128 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.393095 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6p9\" (UniqueName: \"kubernetes.io/projected/7c25eaea-0740-49fa-b603-f15ee861b09a-kube-api-access-zn6p9\") pod \"payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.393294 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.393159 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c25eaea-0740-49fa-b603-f15ee861b09a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.495095 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.495059 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6p9\" (UniqueName: \"kubernetes.io/projected/7c25eaea-0740-49fa-b603-f15ee861b09a-kube-api-access-zn6p9\") pod \"payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.495273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.495115 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c25eaea-0740-49fa-b603-f15ee861b09a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.496142 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.496119 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c25eaea-0740-49fa-b603-f15ee861b09a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.518391 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.518357 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6p9\" (UniqueName: \"kubernetes.io/projected/7c25eaea-0740-49fa-b603-f15ee861b09a-kube-api-access-zn6p9\") pod \"payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.622938 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.622909 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:16.934856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:16.934819 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w"] Nov 03 23:02:16.935862 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:02:16.935838 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c25eaea_0740_49fa_b603_f15ee861b09a.slice/crio-00f0ccba8157ca086b99edf7668ec938fc549f58e26689f090acd6775149f3ff WatchSource:0}: Error finding container 00f0ccba8157ca086b99edf7668ec938fc549f58e26689f090acd6775149f3ff: Status 404 returned error can't find the container with id 00f0ccba8157ca086b99edf7668ec938fc549f58e26689f090acd6775149f3ff Nov 03 23:02:17.569670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:17.569639 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:17.663491 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:17.663455 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" event={"ID":"7c25eaea-0740-49fa-b603-f15ee861b09a","Type":"ContainerStarted","Data":"d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad"} Nov 03 23:02:17.663642 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:17.663534 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" event={"ID":"7c25eaea-0740-49fa-b603-f15ee861b09a","Type":"ContainerStarted","Data":"00f0ccba8157ca086b99edf7668ec938fc549f58e26689f090acd6775149f3ff"} Nov 03 23:02:17.663642 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:17.663607 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:17.705269 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:17.705216 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" podStartSLOduration=1.705202154 podStartE2EDuration="1.705202154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:17.703517344 +0000 UTC m=+2959.132853335" watchObservedRunningTime="2026-11-03 23:02:17.705202154 +0000 UTC m=+2959.134538144" Nov 03 23:02:19.637247 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:19.637213 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:02:24.522509 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.522471 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg"] Nov 03 23:02:24.526021 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.526002 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.547453 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.547428 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3920cb-openshift-default-dockercfg-259xz\"" Nov 03 23:02:24.567537 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.567483 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg"] Nov 03 23:02:24.648827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.648794 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-credential-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649025 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.648843 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-certs\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649073 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.649011 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9vrk\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-kube-api-access-s9vrk\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649124 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.649087 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-token\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649173 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.649145 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649241 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.649230 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-data\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649281 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.649266 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0004e781-c6c0-4815-80a5-9248dcc3fd30-istiod-ca-cert\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649321 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.649296 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-podinfo\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.649452 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.649429 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-envoy\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.750593 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.750553 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-data\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.750822 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.750600 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0004e781-c6c0-4815-80a5-9248dcc3fd30-istiod-ca-cert\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.750822 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.750631 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-podinfo\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.750822 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.750692 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-envoy\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751079 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751050 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-data\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751275 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751159 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-credential-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751382 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751313 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-certs\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751392 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9vrk\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-kube-api-access-s9vrk\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751392 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0004e781-c6c0-4815-80a5-9248dcc3fd30-istiod-ca-cert\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751560 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751463 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-credential-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751560 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751474 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-token\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751681 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751576 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751681 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751627 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-certs\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.751944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.751924 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.753163 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.753137 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-envoy\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.754118 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.754095 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-podinfo\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.775161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.775098 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-token\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.775343 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.775320 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9vrk\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-kube-api-access-s9vrk\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:24.850753 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:24.844553 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:25.369118 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.369085 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg"] Nov 03 23:02:25.369240 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:02:25.369115 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0004e781_c6c0_4815_80a5_9248dcc3fd30.slice/crio-f448aa9c7ce75e11e0920b1e0161d1c119919c8e23af72d9fb9f45dd6f0cb731 WatchSource:0}: Error finding container f448aa9c7ce75e11e0920b1e0161d1c119919c8e23af72d9fb9f45dd6f0cb731: Status 404 returned error can't find the container with id f448aa9c7ce75e11e0920b1e0161d1c119919c8e23af72d9fb9f45dd6f0cb731 Nov 03 23:02:25.372491 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.372454 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:02:25.372623 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.372533 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:02:25.372623 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.372578 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:02:25.729288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.729198 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" event={"ID":"0004e781-c6c0-4815-80a5-9248dcc3fd30","Type":"ContainerStarted","Data":"af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08"} Nov 03 23:02:25.729288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.729254 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" event={"ID":"0004e781-c6c0-4815-80a5-9248dcc3fd30","Type":"ContainerStarted","Data":"f448aa9c7ce75e11e0920b1e0161d1c119919c8e23af72d9fb9f45dd6f0cb731"} Nov 03 23:02:25.786173 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.786035 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podStartSLOduration=1.786014677 podStartE2EDuration="1.786014677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:25.785845914 +0000 UTC m=+2967.215181920" watchObservedRunningTime="2026-11-03 23:02:25.786014677 +0000 UTC m=+2967.215350668" Nov 03 23:02:25.850750 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:25.845805 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:26.853739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:26.852906 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 23:02:26.853739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:26.852975 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 23:02:27.856738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:27.846090 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 23:02:27.856738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:27.846154 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 23:02:28.674981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:28.674948 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:02:28.854739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:28.846087 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 23:02:28.854739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:28.846158 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 23:02:29.854741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:29.849401 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 23:02:29.854741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:29.849470 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 23:02:30.851254 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:30.850121 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 23:02:30.851254 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:30.850193 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 23:02:31.862321 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:31.862282 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 03 23:02:31.863009 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:31.862975 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 03 23:02:32.190145 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:02:32.190049 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:02:32.851367 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:32.851337 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:32.851777 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:32.851754 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:32.852350 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:32.852330 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:02:41.828282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:41.828245 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48"] Nov 03 23:02:41.828865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:41.828554 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" podUID="971c6d57-a245-4ffc-8573-d70e30757ef5" containerName="payload-pre-processing" containerID="cri-o://d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251" gracePeriod=30 Nov 03 23:02:42.304227 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.304150 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:42.467555 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.467519 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971c6d57-a245-4ffc-8573-d70e30757ef5-kube-api-access-2dclz\" (UniqueName: \"kubernetes.io/projected/971c6d57-a245-4ffc-8573-d70e30757ef5-kube-api-access-2dclz\") pod \"971c6d57-a245-4ffc-8573-d70e30757ef5\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " Nov 03 23:02:42.467871 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.467597 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971c6d57-a245-4ffc-8573-d70e30757ef5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971c6d57-a245-4ffc-8573-d70e30757ef5-plugins-config-volume\") pod \"971c6d57-a245-4ffc-8573-d70e30757ef5\" (UID: \"971c6d57-a245-4ffc-8573-d70e30757ef5\") " Nov 03 23:02:42.468020 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.467997 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971c6d57-a245-4ffc-8573-d70e30757ef5-plugins-config-volume" pod "971c6d57-a245-4ffc-8573-d70e30757ef5" (UID: "971c6d57-a245-4ffc-8573-d70e30757ef5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:42.468251 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.468233 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971c6d57-a245-4ffc-8573-d70e30757ef5-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:42.470136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.470109 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971c6d57-a245-4ffc-8573-d70e30757ef5-kube-api-access-2dclz" pod "971c6d57-a245-4ffc-8573-d70e30757ef5" (UID: "971c6d57-a245-4ffc-8573-d70e30757ef5"). InnerVolumeSpecName "kube-api-access-2dclz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:42.589133 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.588974 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dclz\" (UniqueName: \"kubernetes.io/projected/971c6d57-a245-4ffc-8573-d70e30757ef5-kube-api-access-2dclz\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:42.913206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.913175 4625 generic.go:356] "Generic (PLEG): container finished" podID="971c6d57-a245-4ffc-8573-d70e30757ef5" containerID="d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251" exitCode=0 Nov 03 23:02:42.913599 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.913218 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" Nov 03 23:02:42.913599 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.913255 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" event={"ID":"971c6d57-a245-4ffc-8573-d70e30757ef5","Type":"ContainerDied","Data":"d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251"} Nov 03 23:02:42.913599 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.913313 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48" event={"ID":"971c6d57-a245-4ffc-8573-d70e30757ef5","Type":"ContainerDied","Data":"f06ad81a36a093c0eb1a6ffb9b84deec4b995cda69da46f1e9252207cb20bf6d"} Nov 03 23:02:42.913599 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.913336 4625 scope.go:122] "RemoveContainer" containerID="d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251" Nov 03 23:02:42.924018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.924000 4625 scope.go:122] "RemoveContainer" containerID="d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251" Nov 03 23:02:42.924301 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:02:42.924273 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251\": container with ID starting with d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251 not found: ID does not exist" containerID="d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251" Nov 03 23:02:42.924438 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.924300 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251"} err="failed to get container status \"d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251\": rpc error: code = NotFound desc = could not find container \"d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251\": container with ID starting with d05af6a657a07d1f4729d0b77d47f40964af74366657e9bcd77499db2716e251 not found: ID does not exist" Nov 03 23:02:42.967724 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.967681 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48"] Nov 03 23:02:42.980779 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:42.980724 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-cfdc97c95-pfc48"] Nov 03 23:02:44.729667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:44.729630 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971c6d57-a245-4ffc-8573-d70e30757ef5" path="/var/lib/kubelet/pods/971c6d57-a245-4ffc-8573-d70e30757ef5/volumes" Nov 03 23:02:49.220901 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:49.220867 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk"] Nov 03 23:02:49.221377 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:49.221096 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" containerID="cri-o://74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be" gracePeriod=30 Nov 03 23:02:50.631286 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:50.631247 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:02:50.631662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:50.631305 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:02:54.512251 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.512225 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 23:02:54.654057 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654025 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-credential-socket\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654057 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654068 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-envoy\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654336 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654088 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-certs\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654336 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654136 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istiod-ca-cert\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654336 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654277 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-token\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-token\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654521 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654348 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-credential-socket" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:54.654521 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654399 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-kube-api-access-pnq5n\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-kube-api-access-pnq5n\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654521 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654432 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-socket\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654521 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654437 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-certs" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:54.654521 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654509 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-data\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654851 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654543 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-podinfo\") pod \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\" (UID: \"7a608e71-3ef4-4f7e-9e0c-e48c8e935037\") " Nov 03 23:02:54.654851 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654772 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-socket" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:54.654972 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.654935 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istiod-ca-cert" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:54.655054 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.655023 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-data" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:54.655538 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.655512 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.655670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.655541 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.655670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.655558 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.655670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.655574 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.655670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.655587 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.656625 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.656594 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-envoy" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:54.657000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.656945 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-token" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:54.657000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.656981 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-kube-api-access-pnq5n" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "kube-api-access-pnq5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:54.657463 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.657439 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-podinfo" pod "7a608e71-3ef4-4f7e-9e0c-e48c8e935037" (UID: "7a608e71-3ef4-4f7e-9e0c-e48c8e935037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:02:54.756578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.756500 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.756578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.756526 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnq5n\" (UniqueName: \"kubernetes.io/projected/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-kube-api-access-pnq5n\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.756578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.756537 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.756578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:54.756545 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a608e71-3ef4-4f7e-9e0c-e48c8e935037-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:02:55.034711 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.034619 4625 generic.go:356] "Generic (PLEG): container finished" podID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerID="74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be" exitCode=0 Nov 03 23:02:55.034711 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.034677 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" Nov 03 23:02:55.034941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.034706 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" event={"ID":"7a608e71-3ef4-4f7e-9e0c-e48c8e935037","Type":"ContainerDied","Data":"74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be"} Nov 03 23:02:55.034941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.034778 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk" event={"ID":"7a608e71-3ef4-4f7e-9e0c-e48c8e935037","Type":"ContainerDied","Data":"92244396bd25ec1f67f5c6b36f752c0496e4e4e685166337d44af743269d3c91"} Nov 03 23:02:55.034941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.034805 4625 scope.go:122] "RemoveContainer" containerID="74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be" Nov 03 23:02:55.048001 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.047978 4625 scope.go:122] "RemoveContainer" containerID="31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e" Nov 03 23:02:55.060845 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.060704 4625 scope.go:122] "RemoveContainer" containerID="74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be" Nov 03 23:02:55.061322 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:02:55.061297 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be\": container with ID starting with 74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be not found: ID does not exist" containerID="74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be" Nov 03 23:02:55.061436 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.061330 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be"} err="failed to get container status \"74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be\": rpc error: code = NotFound desc = could not find container \"74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be\": container with ID starting with 74b3c4f26db01cb1bf7c03ae03c11257274af24ff0002330040adb1a9ea5b8be not found: ID does not exist" Nov 03 23:02:55.061436 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.061353 4625 scope.go:122] "RemoveContainer" containerID="31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e" Nov 03 23:02:55.061641 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:02:55.061624 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e\": container with ID starting with 31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e not found: ID does not exist" containerID="31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e" Nov 03 23:02:55.061700 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.061653 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e"} err="failed to get container status \"31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e\": rpc error: code = NotFound desc = could not find container \"31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e\": container with ID starting with 31ffe7c91e8f754d05663a1c038b89317958d9a01e65fa74b6d6d309fd505c3e not found: ID does not exist" Nov 03 23:02:55.088413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.088379 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk"] Nov 03 23:02:55.102987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:55.102948 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a01f9-openshift-default-bb6d8b97f-wzwlk"] Nov 03 23:02:56.729781 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:02:56.729750 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" path="/var/lib/kubelet/pods/7a608e71-3ef4-4f7e-9e0c-e48c8e935037/volumes" Nov 03 23:03:02.187316 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:03:02.187274 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:03:03.926998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:03.926971 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:03:03.972211 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:03.972185 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:03:18.008831 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.008745 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79"] Nov 03 23:03:18.009396 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.009001 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" podUID="8a7df9d1-e369-4417-888b-9e4fd6db93a7" containerName="maas-api" containerID="cri-o://0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c" gracePeriod=30 Nov 03 23:03:18.716466 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.713096 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 23:03:18.799959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.799871 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7df9d1-e369-4417-888b-9e4fd6db93a7-kube-api-access-fkqp9\" (UniqueName: \"kubernetes.io/projected/8a7df9d1-e369-4417-888b-9e4fd6db93a7-kube-api-access-fkqp9\") pod \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " Nov 03 23:03:18.799959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.799943 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-maas-api-tls\") pod \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " Nov 03 23:03:18.800190 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.800108 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7df9d1-e369-4417-888b-9e4fd6db93a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7df9d1-e369-4417-888b-9e4fd6db93a7-openshift-service-ca\") pod \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " Nov 03 23:03:18.800251 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.800191 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-metrics-certs\") pod \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\" (UID: \"8a7df9d1-e369-4417-888b-9e4fd6db93a7\") " Nov 03 23:03:18.800972 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.800629 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7df9d1-e369-4417-888b-9e4fd6db93a7-openshift-service-ca" pod "8a7df9d1-e369-4417-888b-9e4fd6db93a7" (UID: "8a7df9d1-e369-4417-888b-9e4fd6db93a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:18.800972 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.800943 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7df9d1-e369-4417-888b-9e4fd6db93a7-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:18.804764 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.803419 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-maas-api-tls" pod "8a7df9d1-e369-4417-888b-9e4fd6db93a7" (UID: "8a7df9d1-e369-4417-888b-9e4fd6db93a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:18.804764 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.803447 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-metrics-certs" pod "8a7df9d1-e369-4417-888b-9e4fd6db93a7" (UID: "8a7df9d1-e369-4417-888b-9e4fd6db93a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:18.805226 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.805201 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7df9d1-e369-4417-888b-9e4fd6db93a7-kube-api-access-fkqp9" pod "8a7df9d1-e369-4417-888b-9e4fd6db93a7" (UID: "8a7df9d1-e369-4417-888b-9e4fd6db93a7"). InnerVolumeSpecName "kube-api-access-fkqp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:18.921839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.906087 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:18.921839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.906122 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7df9d1-e369-4417-888b-9e4fd6db93a7-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:18.921839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:18.906136 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqp9\" (UniqueName: \"kubernetes.io/projected/8a7df9d1-e369-4417-888b-9e4fd6db93a7-kube-api-access-fkqp9\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:19.324831 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.324797 4625 generic.go:356] "Generic (PLEG): container finished" podID="8a7df9d1-e369-4417-888b-9e4fd6db93a7" containerID="0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c" exitCode=0 Nov 03 23:03:19.325307 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.324845 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" Nov 03 23:03:19.325307 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.324880 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" event={"ID":"8a7df9d1-e369-4417-888b-9e4fd6db93a7","Type":"ContainerDied","Data":"0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c"} Nov 03 23:03:19.325307 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.324949 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" event={"ID":"8a7df9d1-e369-4417-888b-9e4fd6db93a7","Type":"ContainerDied","Data":"28c8500c967ca147b52a15f365f16fd46a842d30a34b9aaa5cae5c0e8fa631a0"} Nov 03 23:03:19.325307 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.324983 4625 scope.go:122] "RemoveContainer" containerID="0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c" Nov 03 23:03:19.336237 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.336212 4625 scope.go:122] "RemoveContainer" containerID="0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c" Nov 03 23:03:19.336550 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:03:19.336530 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c\": container with ID starting with 0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c not found: ID does not exist" containerID="0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c" Nov 03 23:03:19.336662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.336557 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c"} err="failed to get container status \"0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c\": rpc error: code = NotFound desc = could not find container \"0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c\": container with ID starting with 0c044b7475f3d622ceb2b3c6d3c8399eab9c4a89d1f102ac740f906deedcea1c not found: ID does not exist" Nov 03 23:03:19.378408 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.378377 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79"] Nov 03 23:03:19.389574 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:19.389544 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79"] Nov 03 23:03:20.728431 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:20.728393 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7df9d1-e369-4417-888b-9e4fd6db93a7" path="/var/lib/kubelet/pods/8a7df9d1-e369-4417-888b-9e4fd6db93a7/volumes" Nov 03 23:03:21.394627 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:21.394586 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3920cb-798684cddc-gnh79" podUID="8a7df9d1-e369-4417-888b-9e4fd6db93a7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.188:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:03:32.186539 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:03:32.186499 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:03:43.633542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:43.633440 4625 generic.go:356] "Generic (PLEG): container finished" podID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerID="9eb2b8f41e7c5b5c2455b95bad80bfd9f33ad9826d35588a8cb2c35d407e58c2" exitCode=0 Nov 03 23:03:43.633542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:43.633473 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" event={"ID":"45159818-f7b5-45a8-b6b8-47873dfd02da","Type":"ContainerDied","Data":"9eb2b8f41e7c5b5c2455b95bad80bfd9f33ad9826d35588a8cb2c35d407e58c2"} Nov 03 23:03:43.634199 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:43.634026 4625 scope.go:122] "RemoveContainer" containerID="9eb2b8f41e7c5b5c2455b95bad80bfd9f33ad9826d35588a8cb2c35d407e58c2" Nov 03 23:03:43.636182 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:43.636153 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:03:43.636310 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:43.636234 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:03:43.636310 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:43.636280 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:03:44.680737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:44.679950 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" event={"ID":"45159818-f7b5-45a8-b6b8-47873dfd02da","Type":"ContainerStarted","Data":"d2fedc25fab15b3622593c08b3056d0b30edd3bfa7b862940e0a2d5088fd0e6a"} Nov 03 23:03:44.909738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:44.907063 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:03:44.909738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:44.908866 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 23:03:44.909738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:44.908935 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 23:03:45.902000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:45.901966 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 23:03:45.902408 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:45.902016 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.820707 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r"] Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821412 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821425 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821437 4625 cpu_manager.go:414] "containerMap: removing container" podUID="971c6d57-a245-4ffc-8573-d70e30757ef5" containerName="payload-pre-processing" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821442 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971c6d57-a245-4ffc-8573-d70e30757ef5" containerName="payload-pre-processing" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821454 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821459 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821482 4625 cpu_manager.go:414] "containerMap: removing container" podUID="8a7df9d1-e369-4417-888b-9e4fd6db93a7" containerName="maas-api" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821488 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7df9d1-e369-4417-888b-9e4fd6db93a7" containerName="maas-api" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821578 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821586 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="971c6d57-a245-4ffc-8573-d70e30757ef5" containerName="payload-pre-processing" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821592 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a608e71-3ef4-4f7e-9e0c-e48c8e935037" containerName="istio-proxy" Nov 03 23:03:46.823738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.821601 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7df9d1-e369-4417-888b-9e4fd6db93a7" containerName="maas-api" Nov 03 23:03:46.835745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.829895 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:03:46.835745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.830368 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.891738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.869381 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f6dcc0-openshift-default-dockercfg-68vmd\"" Nov 03 23:03:46.903074 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.901337 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 23:03:46.903074 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.901420 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 23:03:46.905214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.905187 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r"] Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.991671 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-podinfo\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.991752 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-credential-socket\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.991865 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmdc\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-kube-api-access-tvmdc\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.991913 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-certs\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.991938 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-socket\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.991968 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d2fe03-81da-460a-ba49-df3afe3eefee-istiod-ca-cert\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.992147 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-data\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.992205 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-envoy\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:46.993748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:46.992305 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-token\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.093809 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-data\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.093869 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-envoy\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.093943 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-token\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.094023 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-podinfo\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.094059 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-credential-socket\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.094144 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmdc\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-kube-api-access-tvmdc\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.094187 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-certs\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.094212 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-socket\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.094243 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d2fe03-81da-460a-ba49-df3afe3eefee-istiod-ca-cert\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.095149 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d2fe03-81da-460a-ba49-df3afe3eefee-istiod-ca-cert\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.095746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.095451 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-data\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.133179 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.133140 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-credential-socket\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.248772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.153141 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-certs\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.248772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.153838 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-socket\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.389802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.341571 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg"] Nov 03 23:03:47.402369 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.402332 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" containerID="cri-o://5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708" gracePeriod=30 Nov 03 23:03:47.402705 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.391319 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmdc\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-kube-api-access-tvmdc\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.407317 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.403307 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-podinfo\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.407454 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.402080 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-envoy\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.443754 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.443444 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-token\") pod \"e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.511073 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.507922 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:47.902567 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.902537 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 23:03:47.902768 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:47.902585 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 23:03:48.007629 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.007601 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r"] Nov 03 23:03:48.008079 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:03:48.008057 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d2fe03_81da_460a_ba49_df3afe3eefee.slice/crio-f19539426f2a8a4e5aeda3e4c8fd28b741c9904718f4863f69ee99e8238954ae WatchSource:0}: Error finding container f19539426f2a8a4e5aeda3e4c8fd28b741c9904718f4863f69ee99e8238954ae: Status 404 returned error can't find the container with id f19539426f2a8a4e5aeda3e4c8fd28b741c9904718f4863f69ee99e8238954ae Nov 03 23:03:48.011914 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.011871 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:03:48.012165 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.012130 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:03:48.012272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.012190 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:03:48.713791 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.713753 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" event={"ID":"26d2fe03-81da-460a-ba49-df3afe3eefee","Type":"ContainerStarted","Data":"fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876"} Nov 03 23:03:48.713987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.713820 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" event={"ID":"26d2fe03-81da-460a-ba49-df3afe3eefee","Type":"ContainerStarted","Data":"f19539426f2a8a4e5aeda3e4c8fd28b741c9904718f4863f69ee99e8238954ae"} Nov 03 23:03:48.778530 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.778481 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podStartSLOduration=2.778467463 podStartE2EDuration="2.778467463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:48.775106998 +0000 UTC m=+3050.204442990" watchObservedRunningTime="2026-11-03 23:03:48.778467463 +0000 UTC m=+3050.207803454" Nov 03 23:03:48.905430 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.905393 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 23:03:48.905627 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:48.905451 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 23:03:49.530998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:49.530964 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:49.551764 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:49.546040 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:49.720774 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:49.720745 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:49.721804 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:49.721784 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:03:49.901660 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:49.901630 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 23:03:49.901898 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:49.901688 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 23:03:50.234055 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:50.233711 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:03:50.234055 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:50.233796 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:03:50.930210 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:50.930180 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:03:50.932244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:50.932221 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:03:52.954862 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:52.954834 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 23:03:53.056059 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.056015 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-token\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-token\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.056256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.056100 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-envoy\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.056256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.056177 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-data\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.056256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.056207 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0425c365-f5bc-404a-9d37-4c0066f138dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425c365-f5bc-404a-9d37-4c0066f138dc-istiod-ca-cert\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.056964 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.056792 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-certs\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.056964 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.056873 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0425c365-f5bc-404a-9d37-4c0066f138dc-istiod-ca-cert" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:53.057182 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.056937 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-socket\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.057182 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057017 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-podinfo\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.057182 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057044 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-kube-api-access-kvfxm\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-kube-api-access-kvfxm\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.057182 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057084 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-credential-socket\") pod \"0425c365-f5bc-404a-9d37-4c0066f138dc\" (UID: \"0425c365-f5bc-404a-9d37-4c0066f138dc\") " Nov 03 23:03:53.057408 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057195 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-certs" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:53.057835 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057634 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.057835 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057660 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425c365-f5bc-404a-9d37-4c0066f138dc-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.057835 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057674 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-data" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:53.057835 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.057798 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-socket" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:53.058422 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.058390 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-credential-socket" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:53.058899 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.058863 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-token" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:53.059047 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.059006 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-envoy" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:53.061159 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.061135 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-podinfo" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:53.061566 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.061545 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-kube-api-access-kvfxm" pod "0425c365-f5bc-404a-9d37-4c0066f138dc" (UID: "0425c365-f5bc-404a-9d37-4c0066f138dc"). InnerVolumeSpecName "kube-api-access-kvfxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:53.159163 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.159113 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.159163 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.159145 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.159163 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.159154 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfxm\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-kube-api-access-kvfxm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.159163 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.159164 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.159163 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.159173 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.159578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.159181 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.159578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.159188 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425c365-f5bc-404a-9d37-4c0066f138dc-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:03:53.771443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.771412 4625 generic.go:356] "Generic (PLEG): container finished" podID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerID="5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708" exitCode=0 Nov 03 23:03:53.771606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.771465 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" event={"ID":"0425c365-f5bc-404a-9d37-4c0066f138dc","Type":"ContainerDied","Data":"5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708"} Nov 03 23:03:53.771606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.771474 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" Nov 03 23:03:53.771606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.771512 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg" event={"ID":"0425c365-f5bc-404a-9d37-4c0066f138dc","Type":"ContainerDied","Data":"d80af6d82016c97afe8b02deabb2eab20aa88b3c8bf04b5313ff93fc0918937a"} Nov 03 23:03:53.771606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.771558 4625 scope.go:122] "RemoveContainer" containerID="5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708" Nov 03 23:03:53.783289 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.783093 4625 scope.go:122] "RemoveContainer" containerID="ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e" Nov 03 23:03:53.794430 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.794404 4625 scope.go:122] "RemoveContainer" containerID="5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708" Nov 03 23:03:53.794746 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:03:53.794702 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708\": container with ID starting with 5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708 not found: ID does not exist" containerID="5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708" Nov 03 23:03:53.794826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.794745 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708"} err="failed to get container status \"5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708\": rpc error: code = NotFound desc = could not find container \"5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708\": container with ID starting with 5acc1ae8582ec21bd26b20fbacbf33520a245ac773de926e18825cfea2852708 not found: ID does not exist" Nov 03 23:03:53.794826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.794762 4625 scope.go:122] "RemoveContainer" containerID="ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e" Nov 03 23:03:53.795039 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:03:53.795022 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e\": container with ID starting with ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e not found: ID does not exist" containerID="ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e" Nov 03 23:03:53.795088 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.795044 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e"} err="failed to get container status \"ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e\": rpc error: code = NotFound desc = could not find container \"ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e\": container with ID starting with ea7231758ff9f5e61bc130d9cc2ad6d6df6de452f7fc312d5393c83a61245f5e not found: ID does not exist" Nov 03 23:03:53.872962 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.872928 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg"] Nov 03 23:03:53.885455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:53.885427 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-b4dkg"] Nov 03 23:03:54.729505 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:03:54.729468 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" path="/var/lib/kubelet/pods/0425c365-f5bc-404a-9d37-4c0066f138dc/volumes" Nov 03 23:04:02.187256 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:02.187220 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:04:08.579476 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:08.579440 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg"] Nov 03 23:04:08.580399 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:08.580335 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" containerID="cri-o://af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08" gracePeriod=30 Nov 03 23:04:12.560424 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:12.560375 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w"] Nov 03 23:04:12.561173 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:12.560579 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" podUID="7c25eaea-0740-49fa-b603-f15ee861b09a" containerName="payload-processing" containerID="cri-o://d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad" gracePeriod=30 Nov 03 23:04:12.857107 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:12.857075 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:04:12.927879 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:12.922210 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c25eaea-0740-49fa-b603-f15ee861b09a-kube-api-access-zn6p9\" (UniqueName: \"kubernetes.io/projected/7c25eaea-0740-49fa-b603-f15ee861b09a-kube-api-access-zn6p9\") pod \"7c25eaea-0740-49fa-b603-f15ee861b09a\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " Nov 03 23:04:12.927879 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:12.922344 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c25eaea-0740-49fa-b603-f15ee861b09a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c25eaea-0740-49fa-b603-f15ee861b09a-plugins-config-volume\") pod \"7c25eaea-0740-49fa-b603-f15ee861b09a\" (UID: \"7c25eaea-0740-49fa-b603-f15ee861b09a\") " Nov 03 23:04:12.971458 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:12.951982 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c25eaea-0740-49fa-b603-f15ee861b09a-plugins-config-volume" pod "7c25eaea-0740-49fa-b603-f15ee861b09a" (UID: "7c25eaea-0740-49fa-b603-f15ee861b09a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:13.051772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.034019 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c25eaea-0740-49fa-b603-f15ee861b09a-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.138045 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.124672 4625 generic.go:356] "Generic (PLEG): container finished" podID="7c25eaea-0740-49fa-b603-f15ee861b09a" containerID="d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad" exitCode=0 Nov 03 23:04:13.138045 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.124742 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" event={"ID":"7c25eaea-0740-49fa-b603-f15ee861b09a","Type":"ContainerDied","Data":"d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad"} Nov 03 23:04:13.138045 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.124794 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" event={"ID":"7c25eaea-0740-49fa-b603-f15ee861b09a","Type":"ContainerDied","Data":"00f0ccba8157ca086b99edf7668ec938fc549f58e26689f090acd6775149f3ff"} Nov 03 23:04:13.138045 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.124823 4625 scope.go:122] "RemoveContainer" containerID="d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad" Nov 03 23:04:13.138045 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.125009 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w" Nov 03 23:04:13.138045 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.130812 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c25eaea-0740-49fa-b603-f15ee861b09a-kube-api-access-zn6p9" pod "7c25eaea-0740-49fa-b603-f15ee861b09a" (UID: "7c25eaea-0740-49fa-b603-f15ee861b09a"). InnerVolumeSpecName "kube-api-access-zn6p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:13.163594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.163568 4625 scope.go:122] "RemoveContainer" containerID="d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad" Nov 03 23:04:13.164248 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:13.164217 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad\": container with ID starting with d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad not found: ID does not exist" containerID="d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad" Nov 03 23:04:13.164464 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.164253 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad"} err="failed to get container status \"d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad\": rpc error: code = NotFound desc = could not find container \"d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad\": container with ID starting with d186b3e19dc27ca31194080154894ce19c16ad037ab8b5c3fa2ce8b85639fdad not found: ID does not exist" Nov 03 23:04:13.167756 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.167731 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6p9\" (UniqueName: \"kubernetes.io/projected/7c25eaea-0740-49fa-b603-f15ee861b09a-kube-api-access-zn6p9\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.473207 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.473165 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w"] Nov 03 23:04:13.479449 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.479422 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-696c454488-hc66w"] Nov 03 23:04:13.874545 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.874522 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:04:13.983456 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983360 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-data\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.983456 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983416 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-credential-socket\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.983704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983507 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0004e781-c6c0-4815-80a5-9248dcc3fd30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0004e781-c6c0-4815-80a5-9248dcc3fd30-istiod-ca-cert\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.983704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983582 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-podinfo\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.983704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983683 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-token\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-token\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.983944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983806 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-socket\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.983944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983851 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-data" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.983944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983871 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-certs\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.983944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983859 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-credential-socket" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.984150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.983946 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-kube-api-access-s9vrk\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-kube-api-access-s9vrk\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.984150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984027 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-envoy\") pod \"0004e781-c6c0-4815-80a5-9248dcc3fd30\" (UID: \"0004e781-c6c0-4815-80a5-9248dcc3fd30\") " Nov 03 23:04:13.984150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984110 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-socket" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.984295 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984252 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-certs" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.984747 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984683 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0004e781-c6c0-4815-80a5-9248dcc3fd30-istiod-ca-cert" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:13.984894 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984842 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.984894 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984864 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.984894 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984878 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.985058 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.984894 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.986355 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.986325 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-podinfo" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:04:13.986486 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.986439 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-kube-api-access-s9vrk" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "kube-api-access-s9vrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:13.986603 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.986578 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-token" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:13.986650 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:13.986606 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-envoy" pod "0004e781-c6c0-4815-80a5-9248dcc3fd30" (UID: "0004e781-c6c0-4815-80a5-9248dcc3fd30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:14.086769 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.086695 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0004e781-c6c0-4815-80a5-9248dcc3fd30-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.086769 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.086762 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.086769 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.086771 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.087018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.086779 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9vrk\" (UniqueName: \"kubernetes.io/projected/0004e781-c6c0-4815-80a5-9248dcc3fd30-kube-api-access-s9vrk\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.087018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.086790 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0004e781-c6c0-4815-80a5-9248dcc3fd30-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.135673 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.135635 4625 generic.go:356] "Generic (PLEG): container finished" podID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerID="af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08" exitCode=0 Nov 03 23:04:14.135915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.135725 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" Nov 03 23:04:14.135915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.135756 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" event={"ID":"0004e781-c6c0-4815-80a5-9248dcc3fd30","Type":"ContainerDied","Data":"af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08"} Nov 03 23:04:14.135915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.135815 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg" event={"ID":"0004e781-c6c0-4815-80a5-9248dcc3fd30","Type":"ContainerDied","Data":"f448aa9c7ce75e11e0920b1e0161d1c119919c8e23af72d9fb9f45dd6f0cb731"} Nov 03 23:04:14.135915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.135838 4625 scope.go:122] "RemoveContainer" containerID="af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08" Nov 03 23:04:14.148170 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.148112 4625 scope.go:122] "RemoveContainer" containerID="af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08" Nov 03 23:04:14.148493 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:14.148464 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08\": container with ID starting with af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08 not found: ID does not exist" containerID="af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08" Nov 03 23:04:14.148744 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.148504 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08"} err="failed to get container status \"af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08\": rpc error: code = NotFound desc = could not find container \"af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08\": container with ID starting with af561de28dd33a354f5ee134f2f2faee68c64ae251f3d6ec3abf11dff2b93e08 not found: ID does not exist" Nov 03 23:04:14.190800 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.190765 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg"] Nov 03 23:04:14.206479 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.206440 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-5x4gg"] Nov 03 23:04:14.728928 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.728900 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" path="/var/lib/kubelet/pods/0004e781-c6c0-4815-80a5-9248dcc3fd30/volumes" Nov 03 23:04:14.729306 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:14.729292 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c25eaea-0740-49fa-b603-f15ee861b09a" path="/var/lib/kubelet/pods/7c25eaea-0740-49fa-b603-f15ee861b09a/volumes" Nov 03 23:04:20.078593 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:20.078556 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l"] Nov 03 23:04:20.079103 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:20.078866 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" containerID="cri-o://34e7ed06c0529e1d5ae3a07264addb49ba2995187df83dce301d070157438eef" gracePeriod=30 Nov 03 23:04:21.031111 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:21.031070 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:04:21.031293 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:21.031154 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:04:25.358282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.358207 4625 generic.go:356] "Generic (PLEG): container finished" podID="c506e01f-7155-4b48-8f64-50cbaef69332" containerID="34e7ed06c0529e1d5ae3a07264addb49ba2995187df83dce301d070157438eef" exitCode=0 Nov 03 23:04:25.358697 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.358412 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" event={"ID":"c506e01f-7155-4b48-8f64-50cbaef69332","Type":"ContainerDied","Data":"34e7ed06c0529e1d5ae3a07264addb49ba2995187df83dce301d070157438eef"} Nov 03 23:04:25.358697 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.358454 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" event={"ID":"c506e01f-7155-4b48-8f64-50cbaef69332","Type":"ContainerDied","Data":"eb64fe78444f810ac940cd3fd74224e2893056f25c64963841c02e73ce21569e"} Nov 03 23:04:25.358697 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.358471 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb64fe78444f810ac940cd3fd74224e2893056f25c64963841c02e73ce21569e" Nov 03 23:04:25.358697 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.358488 4625 scope.go:122] "RemoveContainer" containerID="a595c912d64e188415412768e14d46ed825c590ae9a0707b46db45b2684eb935" Nov 03 23:04:25.364917 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.364895 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.441572 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-kube-api-access-zhvzp\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-kube-api-access-zhvzp\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.441640 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-certs\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.441679 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-istio-token\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-istio-token\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.441759 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-data\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.441839 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-envoy\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.442019 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-credential-socket\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.442054 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c506e01f-7155-4b48-8f64-50cbaef69332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c506e01f-7155-4b48-8f64-50cbaef69332-istiod-ca-cert\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.442101 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-socket\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.442272 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c506e01f-7155-4b48-8f64-50cbaef69332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c506e01f-7155-4b48-8f64-50cbaef69332-istio-podinfo\") pod \"c506e01f-7155-4b48-8f64-50cbaef69332\" (UID: \"c506e01f-7155-4b48-8f64-50cbaef69332\") " Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.442999 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-credential-socket" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:25.443127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.443076 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c506e01f-7155-4b48-8f64-50cbaef69332-istiod-ca-cert" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:25.443958 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.443295 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-socket" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:25.443958 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.443480 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-data" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:25.445448 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.445417 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-certs" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:25.447122 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.447090 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c506e01f-7155-4b48-8f64-50cbaef69332-istio-podinfo" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:04:25.447566 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.447540 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-envoy" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:25.449094 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.449068 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-istio-token" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:25.449185 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.449162 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-kube-api-access-zhvzp" pod "c506e01f-7155-4b48-8f64-50cbaef69332" (UID: "c506e01f-7155-4b48-8f64-50cbaef69332"). InnerVolumeSpecName "kube-api-access-zhvzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:25.544594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544562 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvzp\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-kube-api-access-zhvzp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544590 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544600 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c506e01f-7155-4b48-8f64-50cbaef69332-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544609 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544616 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544624 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544632 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c506e01f-7155-4b48-8f64-50cbaef69332-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544640 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c506e01f-7155-4b48-8f64-50cbaef69332-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:25.544951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:25.544647 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c506e01f-7155-4b48-8f64-50cbaef69332-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:26.368093 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.368060 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l" Nov 03 23:04:26.464853 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.464808 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l"] Nov 03 23:04:26.467668 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.467646 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4b994-openshift-default-556f895f89-lpg4l"] Nov 03 23:04:26.650705 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.650628 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2"] Nov 03 23:04:26.651627 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651606 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" Nov 03 23:04:26.651690 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651632 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" Nov 03 23:04:26.651690 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651654 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" Nov 03 23:04:26.651690 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651662 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" Nov 03 23:04:26.651817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651695 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" Nov 03 23:04:26.651817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651703 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" Nov 03 23:04:26.651817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651755 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7c25eaea-0740-49fa-b603-f15ee861b09a" containerName="payload-processing" Nov 03 23:04:26.651817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651763 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c25eaea-0740-49fa-b603-f15ee861b09a" containerName="payload-processing" Nov 03 23:04:26.651941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651918 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" Nov 03 23:04:26.651941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651930 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" Nov 03 23:04:26.652006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651945 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0004e781-c6c0-4815-80a5-9248dcc3fd30" containerName="istio-proxy" Nov 03 23:04:26.652006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.651961 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c25eaea-0740-49fa-b603-f15ee861b09a" containerName="payload-processing" Nov 03 23:04:26.657422 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.657399 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.677621 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.673879 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f6dcc0-serving-cert\"" Nov 03 23:04:26.677621 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.676428 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2"] Nov 03 23:04:26.733470 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.733440 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" path="/var/lib/kubelet/pods/c506e01f-7155-4b48-8f64-50cbaef69332/volumes" Nov 03 23:04:26.762393 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.762360 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.762538 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.762509 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.762696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.762677 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wbt\" (UniqueName: \"kubernetes.io/projected/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-kube-api-access-s5wbt\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.762800 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.762754 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-metrics-certs\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.869737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.867831 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.869737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.868040 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.869737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.868265 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wbt\" (UniqueName: \"kubernetes.io/projected/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-kube-api-access-s5wbt\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.869737 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.868313 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-metrics-certs\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.886740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.872777 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-metrics-certs\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.886740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.875862 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:26.886740 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:26.877092 4625 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-serving-cert: secret "maas-api-e2e-ipp-b-f6dcc0-serving-cert" not found Nov 03 23:04:26.886740 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:26.877172 4625 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls podName:2d07e94e-9f07-4af8-b8fb-1d316bd5705f nodeName:}" failed. No retries permitted until 2026-11-03 23:04:27.377153142 +0000 UTC m=+3088.806489129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls") pod "maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" (UID: "2d07e94e-9f07-4af8-b8fb-1d316bd5705f") : secret "maas-api-e2e-ipp-b-f6dcc0-serving-cert" not found Nov 03 23:04:26.901523 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:26.901448 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wbt\" (UniqueName: \"kubernetes.io/projected/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-kube-api-access-s5wbt\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:27.478190 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:27.478151 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:27.480882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:27.480853 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:27.577279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:27.577248 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:27.889989 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:27.889959 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2"] Nov 03 23:04:27.891611 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:04:27.891580 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d07e94e_9f07_4af8_b8fb_1d316bd5705f.slice/crio-826de72abecfbfcb4aec9f23ef1d782924d3bdf814ef0902e2c3dff16ac39744 WatchSource:0}: Error finding container 826de72abecfbfcb4aec9f23ef1d782924d3bdf814ef0902e2c3dff16ac39744: Status 404 returned error can't find the container with id 826de72abecfbfcb4aec9f23ef1d782924d3bdf814ef0902e2c3dff16ac39744 Nov 03 23:04:28.386195 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.386154 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" event={"ID":"2d07e94e-9f07-4af8-b8fb-1d316bd5705f","Type":"ContainerStarted","Data":"9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1"} Nov 03 23:04:28.386390 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.386248 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" event={"ID":"2d07e94e-9f07-4af8-b8fb-1d316bd5705f","Type":"ContainerStarted","Data":"826de72abecfbfcb4aec9f23ef1d782924d3bdf814ef0902e2c3dff16ac39744"} Nov 03 23:04:28.386390 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.386271 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:28.457743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.456340 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" podStartSLOduration=2.091833655 podStartE2EDuration="2.456316076s" totalImagesPullingTime="364.482421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:28.439893657 +0000 UTC m=+3089.869229653" watchObservedRunningTime="2026-11-03 23:04:28.456316076 +0000 UTC m=+3089.885652084" Nov 03 23:04:28.840786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.840750 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t"] Nov 03 23:04:28.841724 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.841691 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" Nov 03 23:04:28.841790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.841732 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" Nov 03 23:04:28.841790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.841759 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" Nov 03 23:04:28.841790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.841766 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" Nov 03 23:04:28.841973 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.841959 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c506e01f-7155-4b48-8f64-50cbaef69332" containerName="istio-proxy" Nov 03 23:04:28.842047 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.841979 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0425c365-f5bc-404a-9d37-4c0066f138dc" containerName="istio-proxy" Nov 03 23:04:28.846865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.846844 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:28.875519 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.875482 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t"] Nov 03 23:04:28.886533 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.886449 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf"] Nov 03 23:04:28.892008 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.891978 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:28.912905 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:28.910246 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf"] Nov 03 23:04:29.047272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047234 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6qr\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-kube-api-access-9f6qr\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.047438 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047303 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.047438 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047400 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427a0694-0bae-463c-a121-3f84d3e01ada-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.047553 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047531 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.047591 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047561 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.047591 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047583 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.047659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047627 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.047752 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047706 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.047888 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047782 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndt6d\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-kube-api-access-ndt6d\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.047963 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.047903 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.048027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048010 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.048086 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048069 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.048141 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048127 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.048243 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048216 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.048288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048272 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427a0694-0bae-463c-a121-3f84d3e01ada-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.048330 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048306 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.048375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048333 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.048478 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.048460 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.149613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149529 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.149613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149569 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.149885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149624 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.149885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149655 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.149885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149687 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427a0694-0bae-463c-a121-3f84d3e01ada-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.149885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149711 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.149885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149752 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.149885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149782 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.149885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149883 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6qr\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-kube-api-access-9f6qr\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149943 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.149968 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427a0694-0bae-463c-a121-3f84d3e01ada-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150029 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150055 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150058 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150078 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150127 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150162 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150200 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150214 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndt6d\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-kube-api-access-ndt6d\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150325 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.150790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150396 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.150790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150415 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427a0694-0bae-463c-a121-3f84d3e01ada-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.150790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150062 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.150790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150533 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.151066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150828 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.151066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.150900 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.151066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.151027 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.151560 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.151533 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.152804 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.152775 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427a0694-0bae-463c-a121-3f84d3e01ada-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.152933 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.152835 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.153008 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.152993 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.153145 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.153126 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.172988 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.172952 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6qr\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-kube-api-access-9f6qr\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.173150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.173128 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.173238 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.173215 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.173302 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.173257 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndt6d\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-kube-api-access-ndt6d\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.231834 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.231800 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:29.465535 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.465460 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:29.572856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.572160 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf"] Nov 03 23:04:29.572856 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:04:29.572690 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d076d7_0968_4cb1_822e_b1bacd73c7db.slice/crio-cb34bee2426360c49ba8492b372b9fd9205a61b333e42599599a68296d501590 WatchSource:0}: Error finding container cb34bee2426360c49ba8492b372b9fd9205a61b333e42599599a68296d501590: Status 404 returned error can't find the container with id cb34bee2426360c49ba8492b372b9fd9205a61b333e42599599a68296d501590 Nov 03 23:04:29.575503 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.575456 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:04:29.575636 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.575547 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:04:29.575636 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.575592 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:04:29.789830 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.789795 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t"] Nov 03 23:04:29.790345 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:04:29.790300 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427a0694_0bae_463c_a121_3f84d3e01ada.slice/crio-80a8313b5e7cb0c3bfadef79e11da94b30fa37653526992d0ebc607d9e84fbd0 WatchSource:0}: Error finding container 80a8313b5e7cb0c3bfadef79e11da94b30fa37653526992d0ebc607d9e84fbd0: Status 404 returned error can't find the container with id 80a8313b5e7cb0c3bfadef79e11da94b30fa37653526992d0ebc607d9e84fbd0 Nov 03 23:04:29.793478 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.793437 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:04:29.793573 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.793525 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:04:29.793643 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:29.793570 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:04:30.406312 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.406274 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" event={"ID":"427a0694-0bae-463c-a121-3f84d3e01ada","Type":"ContainerStarted","Data":"b1fb56e319e07d70a74146bbf914d86ebd6abdb19bb7478ad687f8e4ff321455"} Nov 03 23:04:30.406711 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.406328 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" event={"ID":"427a0694-0bae-463c-a121-3f84d3e01ada","Type":"ContainerStarted","Data":"80a8313b5e7cb0c3bfadef79e11da94b30fa37653526992d0ebc607d9e84fbd0"} Nov 03 23:04:30.408471 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.408436 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" event={"ID":"a0d076d7-0968-4cb1-822e-b1bacd73c7db","Type":"ContainerStarted","Data":"54a008223cbd4c6fd5eace5753cf7279293c5cf2350870fca0680d0a1edc6701"} Nov 03 23:04:30.408579 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.408518 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" event={"ID":"a0d076d7-0968-4cb1-822e-b1bacd73c7db","Type":"ContainerStarted","Data":"cb34bee2426360c49ba8492b372b9fd9205a61b333e42599599a68296d501590"} Nov 03 23:04:30.468017 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.467987 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:30.472096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.472066 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 23:04:30.472337 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.472302 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 23:04:30.511076 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.511018 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podStartSLOduration=2.510998621 podStartE2EDuration="2.510998621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:30.466118566 +0000 UTC m=+3091.895454582" watchObservedRunningTime="2026-11-03 23:04:30.510998621 +0000 UTC m=+3091.940334613" Nov 03 23:04:30.513854 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:30.513815 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" podStartSLOduration=2.513802036 podStartE2EDuration="2.513802036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:30.512900646 +0000 UTC m=+3091.942236637" watchObservedRunningTime="2026-11-03 23:04:30.513802036 +0000 UTC m=+3091.943138070" Nov 03 23:04:31.232268 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:31.232237 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:31.234496 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:31.234175 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 23:04:31.234496 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:31.234228 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 23:04:31.476750 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:31.476315 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 23:04:31.476750 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:31.476386 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 23:04:32.234177 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:32.234138 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 23:04:32.246674 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:32.234471 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 23:04:32.356042 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:32.355983 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:04:32.472733 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:32.472682 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 23:04:32.473020 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:32.472989 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 23:04:33.233740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:33.232926 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 23:04:33.233740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:33.232981 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 23:04:33.471740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:33.470246 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 23:04:33.471740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:33.470306 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 23:04:34.234766 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:34.232691 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 23:04:34.234766 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:34.232795 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 23:04:34.404984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:34.403822 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:04:34.473745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:34.472099 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 23:04:34.473745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:34.472158 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 23:04:35.232542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:35.232508 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 23:04:35.232781 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:35.232564 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 23:04:35.489822 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:35.470077 4625 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 23:04:35.489822 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:35.470138 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 23:04:36.237004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:36.236968 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:36.237434 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:36.237411 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:36.238126 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:36.238104 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:04:36.477792 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:36.477756 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:36.478213 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:36.478191 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:36.478936 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:36.478919 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:04:43.246814 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:43.246737 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq"] Nov 03 23:04:48.247043 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.246998 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" podUID="701abc1a-231d-42a0-a997-ba3e21759e37" containerName="payload-processing" containerID="cri-o://5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450" gracePeriod=25 Nov 03 23:04:48.535938 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.535909 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 23:04:48.668372 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.668320 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701abc1a-231d-42a0-a997-ba3e21759e37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/701abc1a-231d-42a0-a997-ba3e21759e37-plugins-config-volume\") pod \"701abc1a-231d-42a0-a997-ba3e21759e37\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " Nov 03 23:04:48.668598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.668580 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701abc1a-231d-42a0-a997-ba3e21759e37-kube-api-access-ffvgz\" (UniqueName: \"kubernetes.io/projected/701abc1a-231d-42a0-a997-ba3e21759e37-kube-api-access-ffvgz\") pod \"701abc1a-231d-42a0-a997-ba3e21759e37\" (UID: \"701abc1a-231d-42a0-a997-ba3e21759e37\") " Nov 03 23:04:48.668802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.668777 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701abc1a-231d-42a0-a997-ba3e21759e37-plugins-config-volume" pod "701abc1a-231d-42a0-a997-ba3e21759e37" (UID: "701abc1a-231d-42a0-a997-ba3e21759e37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:48.669225 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.669204 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/701abc1a-231d-42a0-a997-ba3e21759e37-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:48.670875 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.670845 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701abc1a-231d-42a0-a997-ba3e21759e37-kube-api-access-ffvgz" pod "701abc1a-231d-42a0-a997-ba3e21759e37" (UID: "701abc1a-231d-42a0-a997-ba3e21759e37"). InnerVolumeSpecName "kube-api-access-ffvgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:48.700935 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.700904 4625 generic.go:356] "Generic (PLEG): container finished" podID="701abc1a-231d-42a0-a997-ba3e21759e37" containerID="5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450" exitCode=0 Nov 03 23:04:48.701121 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.700947 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" Nov 03 23:04:48.701121 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.700965 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" event={"ID":"701abc1a-231d-42a0-a997-ba3e21759e37","Type":"ContainerDied","Data":"5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450"} Nov 03 23:04:48.701121 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.701045 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq" event={"ID":"701abc1a-231d-42a0-a997-ba3e21759e37","Type":"ContainerDied","Data":"e43f24095af046add983cd1998f4c545430a5551bc407b66a99eff7ed0d533fe"} Nov 03 23:04:48.701121 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.701077 4625 scope.go:122] "RemoveContainer" containerID="5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450" Nov 03 23:04:48.711693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.711672 4625 scope.go:122] "RemoveContainer" containerID="5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450" Nov 03 23:04:48.712012 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:48.711991 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450\": container with ID starting with 5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450 not found: ID does not exist" containerID="5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450" Nov 03 23:04:48.712069 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.712021 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450"} err="failed to get container status \"5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450\": rpc error: code = NotFound desc = could not find container \"5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450\": container with ID starting with 5b25937dd30ec414aeacde5797b68ca32c52b0e2c76173b0c7d743a2d9b9b450 not found: ID does not exist" Nov 03 23:04:48.757990 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.757901 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq"] Nov 03 23:04:48.759281 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.759251 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-553753-cf69c5f4b-q2vgq"] Nov 03 23:04:48.770745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:48.770688 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvgz\" (UniqueName: \"kubernetes.io/projected/701abc1a-231d-42a0-a997-ba3e21759e37-kube-api-access-ffvgz\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:50.732938 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:50.732898 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701abc1a-231d-42a0-a997-ba3e21759e37" path="/var/lib/kubelet/pods/701abc1a-231d-42a0-a997-ba3e21759e37/volumes" Nov 03 23:04:51.166376 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.166339 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j"] Nov 03 23:04:51.166868 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.166837 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" podUID="e0217efd-c387-423c-8e37-a4dc7c89d36e" containerName="maas-api" containerID="cri-o://dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37" gracePeriod=30 Nov 03 23:04:51.512209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.512157 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 23:04:51.612543 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.612501 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0217efd-c387-423c-8e37-a4dc7c89d36e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0217efd-c387-423c-8e37-a4dc7c89d36e-openshift-service-ca\") pod \"e0217efd-c387-423c-8e37-a4dc7c89d36e\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " Nov 03 23:04:51.613705 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.612695 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-metrics-certs\") pod \"e0217efd-c387-423c-8e37-a4dc7c89d36e\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " Nov 03 23:04:51.613705 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.612777 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0217efd-c387-423c-8e37-a4dc7c89d36e-kube-api-access-tw5vs\" (UniqueName: \"kubernetes.io/projected/e0217efd-c387-423c-8e37-a4dc7c89d36e-kube-api-access-tw5vs\") pod \"e0217efd-c387-423c-8e37-a4dc7c89d36e\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " Nov 03 23:04:51.613705 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.613025 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-maas-api-tls\") pod \"e0217efd-c387-423c-8e37-a4dc7c89d36e\" (UID: \"e0217efd-c387-423c-8e37-a4dc7c89d36e\") " Nov 03 23:04:51.614466 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.614437 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0217efd-c387-423c-8e37-a4dc7c89d36e-openshift-service-ca" pod "e0217efd-c387-423c-8e37-a4dc7c89d36e" (UID: "e0217efd-c387-423c-8e37-a4dc7c89d36e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:51.616654 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.616626 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-maas-api-tls" pod "e0217efd-c387-423c-8e37-a4dc7c89d36e" (UID: "e0217efd-c387-423c-8e37-a4dc7c89d36e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:51.617044 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.617018 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-metrics-certs" pod "e0217efd-c387-423c-8e37-a4dc7c89d36e" (UID: "e0217efd-c387-423c-8e37-a4dc7c89d36e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:51.618114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.618086 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0217efd-c387-423c-8e37-a4dc7c89d36e-kube-api-access-tw5vs" pod "e0217efd-c387-423c-8e37-a4dc7c89d36e" (UID: "e0217efd-c387-423c-8e37-a4dc7c89d36e"). InnerVolumeSpecName "kube-api-access-tw5vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:51.715006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.714968 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.715006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.715003 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0217efd-c387-423c-8e37-a4dc7c89d36e-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.715006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.715012 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0217efd-c387-423c-8e37-a4dc7c89d36e-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.715273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.715021 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5vs\" (UniqueName: \"kubernetes.io/projected/e0217efd-c387-423c-8e37-a4dc7c89d36e-kube-api-access-tw5vs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.726451 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.726416 4625 generic.go:356] "Generic (PLEG): container finished" podID="e0217efd-c387-423c-8e37-a4dc7c89d36e" containerID="dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37" exitCode=0 Nov 03 23:04:51.726641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.726533 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" event={"ID":"e0217efd-c387-423c-8e37-a4dc7c89d36e","Type":"ContainerDied","Data":"dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37"} Nov 03 23:04:51.726641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.726537 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" Nov 03 23:04:51.726641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.726575 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j" event={"ID":"e0217efd-c387-423c-8e37-a4dc7c89d36e","Type":"ContainerDied","Data":"06613d031183734d46a68166c61e54e0f33c8642dff7694eb3de040d040c4773"} Nov 03 23:04:51.726641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.726597 4625 scope.go:122] "RemoveContainer" containerID="dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37" Nov 03 23:04:51.742839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.742815 4625 scope.go:122] "RemoveContainer" containerID="dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37" Nov 03 23:04:51.743564 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:04:51.743543 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37\": container with ID starting with dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37 not found: ID does not exist" containerID="dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37" Nov 03 23:04:51.743732 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.743694 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37"} err="failed to get container status \"dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37\": rpc error: code = NotFound desc = could not find container \"dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37\": container with ID starting with dd85cc8accb088d790353c57bc76f67e47b8c691c15927bd562c86cb8d028b37 not found: ID does not exist" Nov 03 23:04:51.784283 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.784220 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j"] Nov 03 23:04:51.792364 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:51.792332 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f48f46-855677c5f7-gqx9j"] Nov 03 23:04:52.730549 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:52.730516 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0217efd-c387-423c-8e37-a4dc7c89d36e" path="/var/lib/kubelet/pods/e0217efd-c387-423c-8e37-a4dc7c89d36e/volumes" Nov 03 23:04:59.379743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:59.378790 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk"] Nov 03 23:04:59.379743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:04:59.379066 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" containerID="cri-o://62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e" gracePeriod=30 Nov 03 23:05:02.189592 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:02.189540 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:05:04.732109 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.731784 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 23:05:04.732109 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.731836 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 23:05:04.776953 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.776914 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 23:05:04.846325 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.846293 4625 generic.go:356] "Generic (PLEG): container finished" podID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerID="fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876" exitCode=0 Nov 03 23:05:04.846529 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.846361 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" event={"ID":"26d2fe03-81da-460a-ba49-df3afe3eefee","Type":"ContainerDied","Data":"fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876"} Nov 03 23:05:04.846980 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.846955 4625 scope.go:122] "RemoveContainer" containerID="fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876" Nov 03 23:05:04.849031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.848696 4625 generic.go:356] "Generic (PLEG): container finished" podID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerID="62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e" exitCode=0 Nov 03 23:05:04.849031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.848899 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" event={"ID":"25766c62-3e88-493d-bfeb-a3e33b86e67c","Type":"ContainerDied","Data":"62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e"} Nov 03 23:05:04.849031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.848948 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" event={"ID":"25766c62-3e88-493d-bfeb-a3e33b86e67c","Type":"ContainerDied","Data":"5936245e921c30e26e74ccfc7945ce918c3f47b20e4bedc8023cd3e230eca996"} Nov 03 23:05:04.849031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.848974 4625 scope.go:122] "RemoveContainer" containerID="62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e" Nov 03 23:05:04.849314 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.849138 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk" Nov 03 23:05:04.849314 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.849122 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:05:04.849414 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.849336 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:05:04.849414 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.849395 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:05:04.866902 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.866876 4625 scope.go:122] "RemoveContainer" containerID="6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd" Nov 03 23:05:04.884133 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.884104 4625 scope.go:122] "RemoveContainer" containerID="62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e" Nov 03 23:05:04.885297 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:04.885264 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e\": container with ID starting with 62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e not found: ID does not exist" containerID="62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e" Nov 03 23:05:04.885416 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.885310 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e"} err="failed to get container status \"62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e\": rpc error: code = NotFound desc = could not find container \"62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e\": container with ID starting with 62e6fa946b805b0b29c8db71d86b0bebcce0d6ab5370e501be5b7f1c7ee0326e not found: ID does not exist" Nov 03 23:05:04.885416 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.885384 4625 scope.go:122] "RemoveContainer" containerID="6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd" Nov 03 23:05:04.885868 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:04.885842 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd\": container with ID starting with 6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd not found: ID does not exist" containerID="6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd" Nov 03 23:05:04.885964 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.885879 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd"} err="failed to get container status \"6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd\": rpc error: code = NotFound desc = could not find container \"6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd\": container with ID starting with 6c16fbf5089b394d65b46170e55e5f66107e2cfb094145251dcfd46051c0a3bd not found: ID does not exist" Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.897749 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-podinfo\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.897823 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-socket\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.897970 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-kube-api-access-77wz7\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-kube-api-access-77wz7\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.898071 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-token\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-token\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.898261 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-credential-socket\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.898425 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-certs\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.898622 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-envoy\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.898848 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25766c62-3e88-493d-bfeb-a3e33b86e67c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25766c62-3e88-493d-bfeb-a3e33b86e67c-istiod-ca-cert\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.899236 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-credential-socket" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.899749 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-data\") pod \"25766c62-3e88-493d-bfeb-a3e33b86e67c\" (UID: \"25766c62-3e88-493d-bfeb-a3e33b86e67c\") " Nov 03 23:05:04.901018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.900533 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:04.901878 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.901849 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-data" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:04.901983 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.901954 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-socket" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:04.903345 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.902167 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25766c62-3e88-493d-bfeb-a3e33b86e67c-istiod-ca-cert" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:04.903345 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.902476 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-certs" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:04.903345 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.903049 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-podinfo" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:04.909970 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.905671 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-envoy" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:04.910179 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.910153 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-kube-api-access-77wz7" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "kube-api-access-77wz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:04.912039 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:04.912001 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-token" pod "25766c62-3e88-493d-bfeb-a3e33b86e67c" (UID: "25766c62-3e88-493d-bfeb-a3e33b86e67c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:05.001616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001582 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.001616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001615 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.001616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001624 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.001961 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001632 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wz7\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-kube-api-access-77wz7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.001961 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001656 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.001961 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001664 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.001961 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001671 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25766c62-3e88-493d-bfeb-a3e33b86e67c-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.001961 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.001679 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25766c62-3e88-493d-bfeb-a3e33b86e67c-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.230018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.229918 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk"] Nov 03 23:05:05.235216 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.235169 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f48f46-openshift-default-5d6fb88f69-l2ttk"] Nov 03 23:05:05.868669 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:05.868631 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" event={"ID":"26d2fe03-81da-460a-ba49-df3afe3eefee","Type":"ContainerStarted","Data":"5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267"} Nov 03 23:05:06.528118 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:06.528010 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:05:06.528364 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:06.528334 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 23:05:06.528463 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:06.528406 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 23:05:06.729885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:06.729856 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" path="/var/lib/kubelet/pods/25766c62-3e88-493d-bfeb-a3e33b86e67c/volumes" Nov 03 23:05:07.528893 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:07.528812 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 23:05:07.528893 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:07.528878 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 23:05:08.528232 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:08.528196 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 23:05:08.528426 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:08.528269 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 23:05:09.529138 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:09.529101 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 23:05:09.529605 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:09.529185 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 23:05:10.528462 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:10.528429 4625 patch_prober.go:29] interesting pod/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 03 23:05:10.528675 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:10.528502 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 03 23:05:10.569089 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:10.569053 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt"] Nov 03 23:05:11.533554 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:11.533522 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:05:11.534025 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:11.534004 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:05:11.534711 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:11.534687 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:05:15.570778 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:15.570727 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" podUID="d2360c3d-31be-4438-aa78-24489e87bb66" containerName="payload-pre-processing" containerID="cri-o://f852378ad4498820bea3638803feabd078d3389d1001dafaa0376b89d0909fae" gracePeriod=25 Nov 03 23:05:15.959870 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:15.959840 4625 generic.go:356] "Generic (PLEG): container finished" podID="d2360c3d-31be-4438-aa78-24489e87bb66" containerID="f852378ad4498820bea3638803feabd078d3389d1001dafaa0376b89d0909fae" exitCode=0 Nov 03 23:05:15.960060 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:15.959911 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" event={"ID":"d2360c3d-31be-4438-aa78-24489e87bb66","Type":"ContainerDied","Data":"f852378ad4498820bea3638803feabd078d3389d1001dafaa0376b89d0909fae"} Nov 03 23:05:15.960060 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:15.959975 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" event={"ID":"d2360c3d-31be-4438-aa78-24489e87bb66","Type":"ContainerDied","Data":"aaca7b999ce63a39d2f6963e0cba5da623dd7c62f690a972143775f6bbe6f452"} Nov 03 23:05:15.960060 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:15.960000 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaca7b999ce63a39d2f6963e0cba5da623dd7c62f690a972143775f6bbe6f452" Nov 03 23:05:15.963211 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:15.963190 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 23:05:16.134917 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:16.134869 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2360c3d-31be-4438-aa78-24489e87bb66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2360c3d-31be-4438-aa78-24489e87bb66-plugins-config-volume\") pod \"d2360c3d-31be-4438-aa78-24489e87bb66\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " Nov 03 23:05:16.135138 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:16.135011 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2360c3d-31be-4438-aa78-24489e87bb66-kube-api-access-fnbvv\" (UniqueName: \"kubernetes.io/projected/d2360c3d-31be-4438-aa78-24489e87bb66-kube-api-access-fnbvv\") pod \"d2360c3d-31be-4438-aa78-24489e87bb66\" (UID: \"d2360c3d-31be-4438-aa78-24489e87bb66\") " Nov 03 23:05:16.135355 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:16.135329 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2360c3d-31be-4438-aa78-24489e87bb66-plugins-config-volume" pod "d2360c3d-31be-4438-aa78-24489e87bb66" (UID: "d2360c3d-31be-4438-aa78-24489e87bb66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:16.135528 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:16.135512 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2360c3d-31be-4438-aa78-24489e87bb66-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:16.137299 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:16.137274 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2360c3d-31be-4438-aa78-24489e87bb66-kube-api-access-fnbvv" pod "d2360c3d-31be-4438-aa78-24489e87bb66" (UID: "d2360c3d-31be-4438-aa78-24489e87bb66"). InnerVolumeSpecName "kube-api-access-fnbvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:16.236580 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:16.236489 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbvv\" (UniqueName: \"kubernetes.io/projected/d2360c3d-31be-4438-aa78-24489e87bb66-kube-api-access-fnbvv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:16.969365 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:16.969325 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt" Nov 03 23:05:17.021439 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:17.021347 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt"] Nov 03 23:05:17.032831 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:17.032797 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-281732-99d75d6c9-xrckt"] Nov 03 23:05:18.581484 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.581445 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb"] Nov 03 23:05:18.582041 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.581694 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" podUID="5f1eacc9-48d4-437b-9421-b451ed4f7ebd" containerName="maas-api" containerID="cri-o://fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194" gracePeriod=30 Nov 03 23:05:18.731348 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.731316 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2360c3d-31be-4438-aa78-24489e87bb66" path="/var/lib/kubelet/pods/d2360c3d-31be-4438-aa78-24489e87bb66/volumes" Nov 03 23:05:18.909951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.909924 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:05:18.984259 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.984221 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-kube-api-access-bjp7d\" (UniqueName: \"kubernetes.io/projected/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-kube-api-access-bjp7d\") pod \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " Nov 03 23:05:18.984486 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.984283 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-openshift-service-ca\") pod \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " Nov 03 23:05:18.985083 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.985049 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-maas-api-tls\") pod \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " Nov 03 23:05:18.985193 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.985122 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-metrics-certs\") pod \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\" (UID: \"5f1eacc9-48d4-437b-9421-b451ed4f7ebd\") " Nov 03 23:05:18.985517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.985493 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-openshift-service-ca" pod "5f1eacc9-48d4-437b-9421-b451ed4f7ebd" (UID: "5f1eacc9-48d4-437b-9421-b451ed4f7ebd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:18.989870 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.989838 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-metrics-certs" pod "5f1eacc9-48d4-437b-9421-b451ed4f7ebd" (UID: "5f1eacc9-48d4-437b-9421-b451ed4f7ebd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:18.989989 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.989865 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-maas-api-tls" pod "5f1eacc9-48d4-437b-9421-b451ed4f7ebd" (UID: "5f1eacc9-48d4-437b-9421-b451ed4f7ebd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:18.989989 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.989980 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-kube-api-access-bjp7d" pod "5f1eacc9-48d4-437b-9421-b451ed4f7ebd" (UID: "5f1eacc9-48d4-437b-9421-b451ed4f7ebd"). InnerVolumeSpecName "kube-api-access-bjp7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:18.993129 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.993104 4625 generic.go:356] "Generic (PLEG): container finished" podID="5f1eacc9-48d4-437b-9421-b451ed4f7ebd" containerID="fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194" exitCode=0 Nov 03 23:05:18.993237 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.993163 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" Nov 03 23:05:18.993307 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.993276 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" event={"ID":"5f1eacc9-48d4-437b-9421-b451ed4f7ebd","Type":"ContainerDied","Data":"fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194"} Nov 03 23:05:18.993376 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.993313 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb" event={"ID":"5f1eacc9-48d4-437b-9421-b451ed4f7ebd","Type":"ContainerDied","Data":"5d9a1206c9e488f8e18680c332f08d2cc036a838b411b8594a4f397565d2f8f0"} Nov 03 23:05:18.993376 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:18.993336 4625 scope.go:122] "RemoveContainer" containerID="fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194" Nov 03 23:05:19.023997 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.023974 4625 scope.go:122] "RemoveContainer" containerID="fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194" Nov 03 23:05:19.024334 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:19.024310 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194\": container with ID starting with fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194 not found: ID does not exist" containerID="fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194" Nov 03 23:05:19.024424 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.024342 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194"} err="failed to get container status \"fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194\": rpc error: code = NotFound desc = could not find container \"fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194\": container with ID starting with fb7ede59ba16c6323ff33ca0586cd22d0da2deb6c7b5865fc5f5aad6ad7da194 not found: ID does not exist" Nov 03 23:05:19.073812 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.073761 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb"] Nov 03 23:05:19.078575 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.078537 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ad693-799c68d7f7-gw4lb"] Nov 03 23:05:19.087509 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.087472 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjp7d\" (UniqueName: \"kubernetes.io/projected/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-kube-api-access-bjp7d\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:19.087509 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.087505 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:19.087509 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.087519 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:19.087787 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:19.087528 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1eacc9-48d4-437b-9421-b451ed4f7ebd-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:20.728775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:20.728707 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1eacc9-48d4-437b-9421-b451ed4f7ebd" path="/var/lib/kubelet/pods/5f1eacc9-48d4-437b-9421-b451ed4f7ebd/volumes" Nov 03 23:05:20.852666 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:20.852626 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp"] Nov 03 23:05:20.852968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:20.852939 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" podUID="b1996cfe-4286-48f4-9db9-93454203b234" containerName="maas-api" containerID="cri-o://4893b6dd1ed7333f8a97f3b561cb4065f6d2fa337d021be29c2063c663556aa8" gracePeriod=30 Nov 03 23:05:20.959413 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:20.959377 4625 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1996cfe_4286_48f4_9db9_93454203b234.slice/crio-conmon-4893b6dd1ed7333f8a97f3b561cb4065f6d2fa337d021be29c2063c663556aa8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1996cfe_4286_48f4_9db9_93454203b234.slice/crio-4893b6dd1ed7333f8a97f3b561cb4065f6d2fa337d021be29c2063c663556aa8.scope\": RecentStats: unable to find data in memory cache]" Nov 03 23:05:21.013325 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.013244 4625 generic.go:356] "Generic (PLEG): container finished" podID="b1996cfe-4286-48f4-9db9-93454203b234" containerID="4893b6dd1ed7333f8a97f3b561cb4065f6d2fa337d021be29c2063c663556aa8" exitCode=0 Nov 03 23:05:21.013325 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.013299 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" event={"ID":"b1996cfe-4286-48f4-9db9-93454203b234","Type":"ContainerDied","Data":"4893b6dd1ed7333f8a97f3b561cb4065f6d2fa337d021be29c2063c663556aa8"} Nov 03 23:05:21.142426 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.142401 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 23:05:21.243987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.243951 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1996cfe-4286-48f4-9db9-93454203b234-kube-api-access-4c29f\" (UniqueName: \"kubernetes.io/projected/b1996cfe-4286-48f4-9db9-93454203b234-kube-api-access-4c29f\") pod \"b1996cfe-4286-48f4-9db9-93454203b234\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " Nov 03 23:05:21.244196 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.244058 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1996cfe-4286-48f4-9db9-93454203b234-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1996cfe-4286-48f4-9db9-93454203b234-openshift-service-ca\") pod \"b1996cfe-4286-48f4-9db9-93454203b234\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " Nov 03 23:05:21.244281 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.244259 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-maas-api-tls\") pod \"b1996cfe-4286-48f4-9db9-93454203b234\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " Nov 03 23:05:21.244393 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.244368 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-metrics-certs\") pod \"b1996cfe-4286-48f4-9db9-93454203b234\" (UID: \"b1996cfe-4286-48f4-9db9-93454203b234\") " Nov 03 23:05:21.244501 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.244435 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1996cfe-4286-48f4-9db9-93454203b234-openshift-service-ca" pod "b1996cfe-4286-48f4-9db9-93454203b234" (UID: "b1996cfe-4286-48f4-9db9-93454203b234"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:21.245021 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.244993 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1996cfe-4286-48f4-9db9-93454203b234-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:21.246790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.246760 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-metrics-certs" pod "b1996cfe-4286-48f4-9db9-93454203b234" (UID: "b1996cfe-4286-48f4-9db9-93454203b234"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:21.246919 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.246896 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-maas-api-tls" pod "b1996cfe-4286-48f4-9db9-93454203b234" (UID: "b1996cfe-4286-48f4-9db9-93454203b234"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:21.246986 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.246916 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1996cfe-4286-48f4-9db9-93454203b234-kube-api-access-4c29f" pod "b1996cfe-4286-48f4-9db9-93454203b234" (UID: "b1996cfe-4286-48f4-9db9-93454203b234"). InnerVolumeSpecName "kube-api-access-4c29f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:21.384739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.381733 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:21.384739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.381770 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1996cfe-4286-48f4-9db9-93454203b234-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:21.384739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:21.381785 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c29f\" (UniqueName: \"kubernetes.io/projected/b1996cfe-4286-48f4-9db9-93454203b234-kube-api-access-4c29f\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:22.058118 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:22.058015 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" event={"ID":"b1996cfe-4286-48f4-9db9-93454203b234","Type":"ContainerDied","Data":"5b668b1aef9074d69fc1e1373bb057f35a5812e0da046011e8218eb835376ea0"} Nov 03 23:05:22.058695 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:22.058649 4625 scope.go:122] "RemoveContainer" containerID="4893b6dd1ed7333f8a97f3b561cb4065f6d2fa337d021be29c2063c663556aa8" Nov 03 23:05:22.059839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:22.059040 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp" Nov 03 23:05:22.257914 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:22.257877 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp"] Nov 03 23:05:22.269300 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:22.269258 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-553753-d74f64f95-cmmlp"] Nov 03 23:05:22.729822 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:22.729785 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1996cfe-4286-48f4-9db9-93454203b234" path="/var/lib/kubelet/pods/b1996cfe-4286-48f4-9db9-93454203b234/volumes" Nov 03 23:05:23.416658 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.416623 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp"] Nov 03 23:05:23.417197 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.416844 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" podUID="4ed4df38-d21b-4b49-a0c8-076f51459320" containerName="maas-api" containerID="cri-o://f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d" gracePeriod=30 Nov 03 23:05:23.843790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.841937 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 23:05:23.919309 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.919272 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed4df38-d21b-4b49-a0c8-076f51459320-kube-api-access-gfpf8\" (UniqueName: \"kubernetes.io/projected/4ed4df38-d21b-4b49-a0c8-076f51459320-kube-api-access-gfpf8\") pod \"4ed4df38-d21b-4b49-a0c8-076f51459320\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " Nov 03 23:05:23.933570 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.926191 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls\") pod \"4ed4df38-d21b-4b49-a0c8-076f51459320\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " Nov 03 23:05:23.933570 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.926340 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-metrics-certs\") pod \"4ed4df38-d21b-4b49-a0c8-076f51459320\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " Nov 03 23:05:23.933570 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.926432 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed4df38-d21b-4b49-a0c8-076f51459320-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed4df38-d21b-4b49-a0c8-076f51459320-openshift-service-ca\") pod \"4ed4df38-d21b-4b49-a0c8-076f51459320\" (UID: \"4ed4df38-d21b-4b49-a0c8-076f51459320\") " Nov 03 23:05:23.933570 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.932104 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed4df38-d21b-4b49-a0c8-076f51459320-openshift-service-ca" pod "4ed4df38-d21b-4b49-a0c8-076f51459320" (UID: "4ed4df38-d21b-4b49-a0c8-076f51459320"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:23.941676 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.940812 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-metrics-certs" pod "4ed4df38-d21b-4b49-a0c8-076f51459320" (UID: "4ed4df38-d21b-4b49-a0c8-076f51459320"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:23.953170 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.953123 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed4df38-d21b-4b49-a0c8-076f51459320-kube-api-access-gfpf8" pod "4ed4df38-d21b-4b49-a0c8-076f51459320" (UID: "4ed4df38-d21b-4b49-a0c8-076f51459320"). InnerVolumeSpecName "kube-api-access-gfpf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:23.968090 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:23.967857 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls" pod "4ed4df38-d21b-4b49-a0c8-076f51459320" (UID: "4ed4df38-d21b-4b49-a0c8-076f51459320"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:24.028882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.028790 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfpf8\" (UniqueName: \"kubernetes.io/projected/4ed4df38-d21b-4b49-a0c8-076f51459320-kube-api-access-gfpf8\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:24.028882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.028830 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:24.028882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.028844 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed4df38-d21b-4b49-a0c8-076f51459320-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:24.028882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.028855 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed4df38-d21b-4b49-a0c8-076f51459320-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:24.075446 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.075408 4625 generic.go:356] "Generic (PLEG): container finished" podID="4ed4df38-d21b-4b49-a0c8-076f51459320" containerID="f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d" exitCode=0 Nov 03 23:05:24.075677 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.075460 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" Nov 03 23:05:24.075677 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.075494 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" event={"ID":"4ed4df38-d21b-4b49-a0c8-076f51459320","Type":"ContainerDied","Data":"f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d"} Nov 03 23:05:24.075677 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.075566 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp" event={"ID":"4ed4df38-d21b-4b49-a0c8-076f51459320","Type":"ContainerDied","Data":"8140c48b6dfb56d0de2b5e05e8ff54ab88c0d6f05d6f9839ae7a4e9c174b98cc"} Nov 03 23:05:24.075677 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.075597 4625 scope.go:122] "RemoveContainer" containerID="f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d" Nov 03 23:05:24.087076 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.087054 4625 scope.go:122] "RemoveContainer" containerID="f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d" Nov 03 23:05:24.087413 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:24.087394 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d\": container with ID starting with f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d not found: ID does not exist" containerID="f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d" Nov 03 23:05:24.087473 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.087425 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d"} err="failed to get container status \"f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d\": rpc error: code = NotFound desc = could not find container \"f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d\": container with ID starting with f384e04944673035b30859690ffd9f15b67b7c397e2ef1f9fe350e9740324f8d not found: ID does not exist" Nov 03 23:05:24.128603 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.128565 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp"] Nov 03 23:05:24.131964 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.131941 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-281732-5cb9bf87cf-vxjsp"] Nov 03 23:05:24.729929 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:24.729836 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed4df38-d21b-4b49-a0c8-076f51459320" path="/var/lib/kubelet/pods/4ed4df38-d21b-4b49-a0c8-076f51459320/volumes" Nov 03 23:05:26.841120 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:26.840493 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf"] Nov 03 23:05:26.841120 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:26.840751 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" containerID="cri-o://54a008223cbd4c6fd5eace5753cf7279293c5cf2350870fca0680d0a1edc6701" gracePeriod=30 Nov 03 23:05:26.846106 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:26.846080 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq"] Nov 03 23:05:26.846639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:26.846566 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" containerID="cri-o://d2fedc25fab15b3622593c08b3056d0b30edd3bfa7b862940e0a2d5088fd0e6a" gracePeriod=30 Nov 03 23:05:26.856190 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:26.856133 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t"] Nov 03 23:05:26.856413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:26.856390 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" containerID="cri-o://b1fb56e319e07d70a74146bbf914d86ebd6abdb19bb7478ad687f8e4ff321455" gracePeriod=30 Nov 03 23:05:31.402456 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.402416 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp"] Nov 03 23:05:31.402873 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.402668 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" podUID="18fe9477-aa04-483c-9d15-39df7f73c174" containerName="maas-api" containerID="cri-o://b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050" gracePeriod=30 Nov 03 23:05:31.786451 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.786415 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 23:05:31.865441 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.865396 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18fe9477-aa04-483c-9d15-39df7f73c174-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fe9477-aa04-483c-9d15-39df7f73c174-openshift-service-ca\") pod \"18fe9477-aa04-483c-9d15-39df7f73c174\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " Nov 03 23:05:31.865619 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.865517 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fe9477-aa04-483c-9d15-39df7f73c174-kube-api-access-mn9jd\" (UniqueName: \"kubernetes.io/projected/18fe9477-aa04-483c-9d15-39df7f73c174-kube-api-access-mn9jd\") pod \"18fe9477-aa04-483c-9d15-39df7f73c174\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " Nov 03 23:05:31.865709 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.865692 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-metrics-certs\") pod \"18fe9477-aa04-483c-9d15-39df7f73c174\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " Nov 03 23:05:31.865817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.865755 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-maas-api-tls\") pod \"18fe9477-aa04-483c-9d15-39df7f73c174\" (UID: \"18fe9477-aa04-483c-9d15-39df7f73c174\") " Nov 03 23:05:31.874792 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.867469 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fe9477-aa04-483c-9d15-39df7f73c174-openshift-service-ca" pod "18fe9477-aa04-483c-9d15-39df7f73c174" (UID: "18fe9477-aa04-483c-9d15-39df7f73c174"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:31.880169 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.880127 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-metrics-certs" pod "18fe9477-aa04-483c-9d15-39df7f73c174" (UID: "18fe9477-aa04-483c-9d15-39df7f73c174"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:31.885022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.884972 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-maas-api-tls" pod "18fe9477-aa04-483c-9d15-39df7f73c174" (UID: "18fe9477-aa04-483c-9d15-39df7f73c174"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:31.887605 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.886834 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fe9477-aa04-483c-9d15-39df7f73c174-kube-api-access-mn9jd" pod "18fe9477-aa04-483c-9d15-39df7f73c174" (UID: "18fe9477-aa04-483c-9d15-39df7f73c174"). InnerVolumeSpecName "kube-api-access-mn9jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.000114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.967817 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn9jd\" (UniqueName: \"kubernetes.io/projected/18fe9477-aa04-483c-9d15-39df7f73c174-kube-api-access-mn9jd\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.000114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.967857 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.000114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.967872 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fe9477-aa04-483c-9d15-39df7f73c174-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.000114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:31.967885 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fe9477-aa04-483c-9d15-39df7f73c174-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.185792 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.176203 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd"] Nov 03 23:05:32.185792 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.176507 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" containerID="cri-o://fa58ea476a64a128da80ceede5cb7b009a07c3f84ce3381bd8bae23892034f85" gracePeriod=30 Nov 03 23:05:32.205453 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.205417 4625 generic.go:356] "Generic (PLEG): container finished" podID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerID="d2fedc25fab15b3622593c08b3056d0b30edd3bfa7b862940e0a2d5088fd0e6a" exitCode=0 Nov 03 23:05:32.205853 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.205830 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" event={"ID":"45159818-f7b5-45a8-b6b8-47873dfd02da","Type":"ContainerDied","Data":"d2fedc25fab15b3622593c08b3056d0b30edd3bfa7b862940e0a2d5088fd0e6a"} Nov 03 23:05:32.206126 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.206108 4625 scope.go:122] "RemoveContainer" containerID="9eb2b8f41e7c5b5c2455b95bad80bfd9f33ad9826d35588a8cb2c35d407e58c2" Nov 03 23:05:32.210776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.210455 4625 generic.go:356] "Generic (PLEG): container finished" podID="18fe9477-aa04-483c-9d15-39df7f73c174" containerID="b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050" exitCode=0 Nov 03 23:05:32.210776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.210558 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" Nov 03 23:05:32.210776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.210664 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" event={"ID":"18fe9477-aa04-483c-9d15-39df7f73c174","Type":"ContainerDied","Data":"b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050"} Nov 03 23:05:32.210776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.210711 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp" event={"ID":"18fe9477-aa04-483c-9d15-39df7f73c174","Type":"ContainerDied","Data":"4908fd62ef41a98078d519afd2f49dca00a267e894e03baa3538a507d8838e16"} Nov 03 23:05:32.218334 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.218003 4625 generic.go:356] "Generic (PLEG): container finished" podID="427a0694-0bae-463c-a121-3f84d3e01ada" containerID="b1fb56e319e07d70a74146bbf914d86ebd6abdb19bb7478ad687f8e4ff321455" exitCode=0 Nov 03 23:05:32.218334 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.218189 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" event={"ID":"427a0694-0bae-463c-a121-3f84d3e01ada","Type":"ContainerDied","Data":"b1fb56e319e07d70a74146bbf914d86ebd6abdb19bb7478ad687f8e4ff321455"} Nov 03 23:05:32.220800 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.220770 4625 generic.go:356] "Generic (PLEG): container finished" podID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerID="54a008223cbd4c6fd5eace5753cf7279293c5cf2350870fca0680d0a1edc6701" exitCode=0 Nov 03 23:05:32.221052 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.221032 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" event={"ID":"a0d076d7-0968-4cb1-822e-b1bacd73c7db","Type":"ContainerDied","Data":"54a008223cbd4c6fd5eace5753cf7279293c5cf2350870fca0680d0a1edc6701"} Nov 03 23:05:32.250106 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:32.250053 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:05:32.275744 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.266495 4625 scope.go:122] "RemoveContainer" containerID="b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050" Nov 03 23:05:32.443790 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.440211 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp"] Nov 03 23:05:32.447802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.445429 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b023a2-6c64666675-49ksp"] Nov 03 23:05:32.456926 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.456779 4625 scope.go:122] "RemoveContainer" containerID="b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050" Nov 03 23:05:32.457178 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:32.457154 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050\": container with ID starting with b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050 not found: ID does not exist" containerID="b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050" Nov 03 23:05:32.457276 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.457188 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050"} err="failed to get container status \"b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050\": rpc error: code = NotFound desc = could not find container \"b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050\": container with ID starting with b8991a3ea26a3efe5e9e05c4815de42c9bccce2f2670e2530d47479d689cb050 not found: ID does not exist" Nov 03 23:05:32.604496 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.604307 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:05:32.635321 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635284 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-credential-socket\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635321 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635327 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-podinfo\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635395 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-certs\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635480 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-socket\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635502 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-token\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-token\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635561 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-envoy\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635586 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-data\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635935 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635615 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-kube-api-access-9f6qr\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-kube-api-access-9f6qr\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.635935 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.635658 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istiod-ca-cert\") pod \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\" (UID: \"a0d076d7-0968-4cb1-822e-b1bacd73c7db\") " Nov 03 23:05:32.636053 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.636029 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istiod-ca-cert" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:32.636244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.636227 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-credential-socket" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.637113 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.637081 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.637243 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.637116 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.638090 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.638059 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-data" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.638449 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.638420 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-socket" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.638530 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.638515 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-certs" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.639235 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.639195 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-envoy" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.639518 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.639494 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-token" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.640344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.640311 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-podinfo" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:32.640794 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.640768 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-kube-api-access-9f6qr" pod "a0d076d7-0968-4cb1-822e-b1bacd73c7db" (UID: "a0d076d7-0968-4cb1-822e-b1bacd73c7db"). InnerVolumeSpecName "kube-api-access-9f6qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.669649 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.664434 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:05:32.674096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.674005 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:05:32.730049 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.730015 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18fe9477-aa04-483c-9d15-39df7f73c174" path="/var/lib/kubelet/pods/18fe9477-aa04-483c-9d15-39df7f73c174/volumes" Nov 03 23:05:32.745375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.744535 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6qr\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-kube-api-access-9f6qr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.745375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.744572 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.745375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.744588 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.745375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.744601 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.745375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.744614 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.745375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.744627 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.745375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.744639 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d076d7-0968-4cb1-822e-b1bacd73c7db-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.845775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.845695 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-token\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-token\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.845792 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45159818-f7b5-45a8-b6b8-47873dfd02da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45159818-f7b5-45a8-b6b8-47873dfd02da-istiod-ca-cert\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.845820 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/427a0694-0bae-463c-a121-3f84d3e01ada-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427a0694-0bae-463c-a121-3f84d3e01ada-istio-podinfo\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.846011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.845850 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427a0694-0bae-463c-a121-3f84d3e01ada-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427a0694-0bae-463c-a121-3f84d3e01ada-istiod-ca-cert\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.846011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.845881 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-podinfo\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.845912 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-certs\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.846011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.845959 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-data\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846373 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846054 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-socket\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846373 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846115 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-credential-socket\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.846373 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846205 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-credential-socket\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846373 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846239 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-certs\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846373 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846291 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-kube-api-access-vnfp5\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-kube-api-access-vnfp5\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846379 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-socket\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.846657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846376 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-certs" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.846657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846441 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-envoy\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.846657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846478 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-envoy\") pod \"45159818-f7b5-45a8-b6b8-47873dfd02da\" (UID: \"45159818-f7b5-45a8-b6b8-47873dfd02da\") " Nov 03 23:05:32.846657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846506 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-istio-token\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-istio-token\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.846657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846554 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-kube-api-access-ndt6d\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-kube-api-access-ndt6d\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.847023 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846580 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427a0694-0bae-463c-a121-3f84d3e01ada-istiod-ca-cert" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:32.847023 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846588 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-data\") pod \"427a0694-0bae-463c-a121-3f84d3e01ada\" (UID: \"427a0694-0bae-463c-a121-3f84d3e01ada\") " Nov 03 23:05:32.847023 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846830 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-data" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.847023 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.846859 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45159818-f7b5-45a8-b6b8-47873dfd02da-istiod-ca-cert" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:32.847255 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847025 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-data" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.847255 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847195 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-socket" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.847590 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847563 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.847685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847594 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.847685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847610 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45159818-f7b5-45a8-b6b8-47873dfd02da-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.847685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847625 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427a0694-0bae-463c-a121-3f84d3e01ada-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.847685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847638 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.847685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.847650 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.849006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.848976 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-podinfo" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:32.849315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.849292 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-credential-socket" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.849516 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.849496 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-socket" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.849710 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.849690 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-credential-socket" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.849925 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.849901 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-certs" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.850347 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.850296 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-envoy" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.851596 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.851574 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-envoy" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.851702 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.851609 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-token" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.851805 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.851753 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-kube-api-access-vnfp5" pod "45159818-f7b5-45a8-b6b8-47873dfd02da" (UID: "45159818-f7b5-45a8-b6b8-47873dfd02da"). InnerVolumeSpecName "kube-api-access-vnfp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.851805 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.851790 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-istio-token" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.852984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.852958 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/427a0694-0bae-463c-a121-3f84d3e01ada-istio-podinfo" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:32.853095 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.853076 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-kube-api-access-ndt6d" pod "427a0694-0bae-463c-a121-3f84d3e01ada" (UID: "427a0694-0bae-463c-a121-3f84d3e01ada"). InnerVolumeSpecName "kube-api-access-ndt6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.948984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.948935 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.948984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.948973 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.948984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.948982 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.948984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.948993 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.948984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949002 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.949443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949011 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfp5\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-kube-api-access-vnfp5\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.949443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949023 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427a0694-0bae-463c-a121-3f84d3e01ada-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.949443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949036 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.949443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949044 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.949443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949052 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndt6d\" (UniqueName: \"kubernetes.io/projected/427a0694-0bae-463c-a121-3f84d3e01ada-kube-api-access-ndt6d\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.949443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949060 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45159818-f7b5-45a8-b6b8-47873dfd02da-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.949443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:32.949067 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427a0694-0bae-463c-a121-3f84d3e01ada-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:33.230783 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.230694 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" Nov 03 23:05:33.230783 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.230734 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t" event={"ID":"427a0694-0bae-463c-a121-3f84d3e01ada","Type":"ContainerDied","Data":"80a8313b5e7cb0c3bfadef79e11da94b30fa37653526992d0ebc607d9e84fbd0"} Nov 03 23:05:33.231000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.230793 4625 scope.go:122] "RemoveContainer" containerID="b1fb56e319e07d70a74146bbf914d86ebd6abdb19bb7478ad687f8e4ff321455" Nov 03 23:05:33.232497 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.232465 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" Nov 03 23:05:33.232632 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.232543 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf" event={"ID":"a0d076d7-0968-4cb1-822e-b1bacd73c7db","Type":"ContainerDied","Data":"cb34bee2426360c49ba8492b372b9fd9205a61b333e42599599a68296d501590"} Nov 03 23:05:33.236013 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.234577 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" event={"ID":"45159818-f7b5-45a8-b6b8-47873dfd02da","Type":"ContainerDied","Data":"079573f5228adbc439bd8e9b04f746e5baa5334de4fa2730801874634f240f6a"} Nov 03 23:05:33.236013 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.234672 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq" Nov 03 23:05:33.269464 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.246267 4625 scope.go:122] "RemoveContainer" containerID="54a008223cbd4c6fd5eace5753cf7279293c5cf2350870fca0680d0a1edc6701" Nov 03 23:05:33.301509 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.301483 4625 scope.go:122] "RemoveContainer" containerID="d2fedc25fab15b3622593c08b3056d0b30edd3bfa7b862940e0a2d5088fd0e6a" Nov 03 23:05:33.315628 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.315603 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf"] Nov 03 23:05:33.329193 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.329164 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-j6vnf"] Nov 03 23:05:33.363089 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.363048 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq"] Nov 03 23:05:33.366386 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.366358 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-224cq"] Nov 03 23:05:33.400183 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.400144 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t"] Nov 03 23:05:33.404523 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:33.404493 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-q767t"] Nov 03 23:05:34.357815 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:34.357775 4625 patch_prober.go:29] interesting pod/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:05:34.358319 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:34.357848 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:05:34.729614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:34.729534 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" path="/var/lib/kubelet/pods/427a0694-0bae-463c-a121-3f84d3e01ada/volumes" Nov 03 23:05:34.730096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:34.730080 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" path="/var/lib/kubelet/pods/45159818-f7b5-45a8-b6b8-47873dfd02da/volumes" Nov 03 23:05:34.730558 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:34.730545 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" path="/var/lib/kubelet/pods/a0d076d7-0968-4cb1-822e-b1bacd73c7db/volumes" Nov 03 23:05:37.275839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.275808 4625 generic.go:356] "Generic (PLEG): container finished" podID="904859f4-8b39-48ed-8289-283eeedec8bd" containerID="fa58ea476a64a128da80ceede5cb7b009a07c3f84ce3381bd8bae23892034f85" exitCode=0 Nov 03 23:05:37.276282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.275882 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" event={"ID":"904859f4-8b39-48ed-8289-283eeedec8bd","Type":"ContainerDied","Data":"fa58ea476a64a128da80ceede5cb7b009a07c3f84ce3381bd8bae23892034f85"} Nov 03 23:05:37.276282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.275961 4625 scope.go:122] "RemoveContainer" containerID="4de94923a3b9b694292f435d1bae87e25d7b9594ef56e173cb54ed100c3f7f90" Nov 03 23:05:37.888179 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.888149 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 23:05:37.918338 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918303 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-istio-token\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-istio-token\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918338 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918344 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-certs\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918411 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-kube-api-access-7p8lh\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-kube-api-access-7p8lh\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918446 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904859f4-8b39-48ed-8289-283eeedec8bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904859f4-8b39-48ed-8289-283eeedec8bd-istiod-ca-cert\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918507 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-data\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918558 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-credential-socket\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918623 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/904859f4-8b39-48ed-8289-283eeedec8bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904859f4-8b39-48ed-8289-283eeedec8bd-istio-podinfo\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918681 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-envoy\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.918869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918768 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-certs" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:37.918869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.918821 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-socket\") pod \"904859f4-8b39-48ed-8289-283eeedec8bd\" (UID: \"904859f4-8b39-48ed-8289-283eeedec8bd\") " Nov 03 23:05:37.919078 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919004 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-credential-socket" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:37.919078 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919020 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-data" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:37.919237 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919215 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-socket" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:37.919344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919320 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904859f4-8b39-48ed-8289-283eeedec8bd-istiod-ca-cert" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:37.919644 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919612 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:37.919644 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919632 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904859f4-8b39-48ed-8289-283eeedec8bd-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:37.919644 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919644 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:37.919879 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919654 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:37.919879 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.919665 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:37.920982 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.920955 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-istio-token" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:37.921270 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.921249 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-kube-api-access-7p8lh" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "kube-api-access-7p8lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:37.921350 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.921249 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-envoy" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:37.921424 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:37.921408 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/904859f4-8b39-48ed-8289-283eeedec8bd-istio-podinfo" pod "904859f4-8b39-48ed-8289-283eeedec8bd" (UID: "904859f4-8b39-48ed-8289-283eeedec8bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:38.020687 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.020594 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904859f4-8b39-48ed-8289-283eeedec8bd-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:38.020687 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.020626 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904859f4-8b39-48ed-8289-283eeedec8bd-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:38.020687 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.020635 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:38.020687 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.020645 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8lh\" (UniqueName: \"kubernetes.io/projected/904859f4-8b39-48ed-8289-283eeedec8bd-kube-api-access-7p8lh\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:38.285043 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.284959 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" Nov 03 23:05:38.285534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.285055 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd" event={"ID":"904859f4-8b39-48ed-8289-283eeedec8bd","Type":"ContainerDied","Data":"bb6d768bc3e7d541581fb0e274166faf5ad42482c116437839cddce647d27051"} Nov 03 23:05:38.285534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.285121 4625 scope.go:122] "RemoveContainer" containerID="fa58ea476a64a128da80ceede5cb7b009a07c3f84ce3381bd8bae23892034f85" Nov 03 23:05:38.341015 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.340957 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd"] Nov 03 23:05:38.350634 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.350602 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-281732-openshift-default-868d787ccc-qgmvd"] Nov 03 23:05:38.729633 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:38.729601 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" path="/var/lib/kubelet/pods/904859f4-8b39-48ed-8289-283eeedec8bd/volumes" Nov 03 23:05:47.904809 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:47.904773 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5"] Nov 03 23:05:47.905329 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:47.905023 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" podUID="3123211a-c11f-4479-be91-2bf9edcb267e" containerName="maas-api" containerID="cri-o://9e75bba24adad5f148842f8ed0cea08656e2374a9387393d1aca01367bfe905f" gracePeriod=30 Nov 03 23:05:48.305543 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.305194 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" podUID="3123211a-c11f-4479-be91-2bf9edcb267e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.136:8443/health\": dial tcp 10.134.0.136:8443: connect: connection refused" Nov 03 23:05:48.390455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.390419 4625 generic.go:356] "Generic (PLEG): container finished" podID="3123211a-c11f-4479-be91-2bf9edcb267e" containerID="9e75bba24adad5f148842f8ed0cea08656e2374a9387393d1aca01367bfe905f" exitCode=0 Nov 03 23:05:48.390629 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.390491 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" event={"ID":"3123211a-c11f-4479-be91-2bf9edcb267e","Type":"ContainerDied","Data":"9e75bba24adad5f148842f8ed0cea08656e2374a9387393d1aca01367bfe905f"} Nov 03 23:05:48.525343 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.525314 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 23:05:48.626071 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.626039 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3123211a-c11f-4479-be91-2bf9edcb267e-kube-api-access-msjfc\" (UniqueName: \"kubernetes.io/projected/3123211a-c11f-4479-be91-2bf9edcb267e-kube-api-access-msjfc\") pod \"3123211a-c11f-4479-be91-2bf9edcb267e\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " Nov 03 23:05:48.626235 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.626077 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3123211a-c11f-4479-be91-2bf9edcb267e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3123211a-c11f-4479-be91-2bf9edcb267e-openshift-service-ca\") pod \"3123211a-c11f-4479-be91-2bf9edcb267e\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " Nov 03 23:05:48.626235 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.626127 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-metrics-certs\") pod \"3123211a-c11f-4479-be91-2bf9edcb267e\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " Nov 03 23:05:48.626386 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.626368 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls\") pod \"3123211a-c11f-4479-be91-2bf9edcb267e\" (UID: \"3123211a-c11f-4479-be91-2bf9edcb267e\") " Nov 03 23:05:48.626776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.626645 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3123211a-c11f-4479-be91-2bf9edcb267e-openshift-service-ca" pod "3123211a-c11f-4479-be91-2bf9edcb267e" (UID: "3123211a-c11f-4479-be91-2bf9edcb267e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:48.626992 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.626969 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3123211a-c11f-4479-be91-2bf9edcb267e-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:48.628786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.628706 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3123211a-c11f-4479-be91-2bf9edcb267e-kube-api-access-msjfc" pod "3123211a-c11f-4479-be91-2bf9edcb267e" (UID: "3123211a-c11f-4479-be91-2bf9edcb267e"). InnerVolumeSpecName "kube-api-access-msjfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:48.628909 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.628876 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls" pod "3123211a-c11f-4479-be91-2bf9edcb267e" (UID: "3123211a-c11f-4479-be91-2bf9edcb267e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:48.629342 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.629320 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-metrics-certs" pod "3123211a-c11f-4479-be91-2bf9edcb267e" (UID: "3123211a-c11f-4479-be91-2bf9edcb267e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:48.729161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.729119 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:48.729161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.729157 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3123211a-c11f-4479-be91-2bf9edcb267e-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:48.729371 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:48.729172 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjfc\" (UniqueName: \"kubernetes.io/projected/3123211a-c11f-4479-be91-2bf9edcb267e-kube-api-access-msjfc\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:49.399625 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:49.399587 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" event={"ID":"3123211a-c11f-4479-be91-2bf9edcb267e","Type":"ContainerDied","Data":"bcf6834d99a89353f70c343445824d46c8366a35f1f5c1af6e09c3100a2412f9"} Nov 03 23:05:49.399625 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:49.399610 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5" Nov 03 23:05:49.400215 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:49.399664 4625 scope.go:122] "RemoveContainer" containerID="9e75bba24adad5f148842f8ed0cea08656e2374a9387393d1aca01367bfe905f" Nov 03 23:05:49.454084 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:49.453995 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5"] Nov 03 23:05:49.465735 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:49.465687 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2a2eca-64b4c88b9d-5lhc5"] Nov 03 23:05:49.992352 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:49.992316 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2"] Nov 03 23:05:49.992574 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:49.992547 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" podUID="2d07e94e-9f07-4af8-b8fb-1d316bd5705f" containerName="maas-api" containerID="cri-o://9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1" gracePeriod=30 Nov 03 23:05:50.260333 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.260260 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:05:50.362827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.359329 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-kube-api-access-s5wbt\" (UniqueName: \"kubernetes.io/projected/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-kube-api-access-s5wbt\") pod \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " Nov 03 23:05:50.362827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.359438 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls\") pod \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " Nov 03 23:05:50.362827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.359498 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-metrics-certs\") pod \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " Nov 03 23:05:50.362827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.359610 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-openshift-service-ca\") pod \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\" (UID: \"2d07e94e-9f07-4af8-b8fb-1d316bd5705f\") " Nov 03 23:05:50.373993 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.373955 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-openshift-service-ca" pod "2d07e94e-9f07-4af8-b8fb-1d316bd5705f" (UID: "2d07e94e-9f07-4af8-b8fb-1d316bd5705f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:50.386507 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.386471 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls" pod "2d07e94e-9f07-4af8-b8fb-1d316bd5705f" (UID: "2d07e94e-9f07-4af8-b8fb-1d316bd5705f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:50.387268 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.387198 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-metrics-certs" pod "2d07e94e-9f07-4af8-b8fb-1d316bd5705f" (UID: "2d07e94e-9f07-4af8-b8fb-1d316bd5705f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:50.402475 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.402443 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-kube-api-access-s5wbt" pod "2d07e94e-9f07-4af8-b8fb-1d316bd5705f" (UID: "2d07e94e-9f07-4af8-b8fb-1d316bd5705f"). InnerVolumeSpecName "kube-api-access-s5wbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:50.421928 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.421897 4625 generic.go:356] "Generic (PLEG): container finished" podID="2d07e94e-9f07-4af8-b8fb-1d316bd5705f" containerID="9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1" exitCode=0 Nov 03 23:05:50.422101 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.422029 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" event={"ID":"2d07e94e-9f07-4af8-b8fb-1d316bd5705f","Type":"ContainerDied","Data":"9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1"} Nov 03 23:05:50.422101 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.422084 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" event={"ID":"2d07e94e-9f07-4af8-b8fb-1d316bd5705f","Type":"ContainerDied","Data":"826de72abecfbfcb4aec9f23ef1d782924d3bdf814ef0902e2c3dff16ac39744"} Nov 03 23:05:50.422226 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.422113 4625 scope.go:122] "RemoveContainer" containerID="9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1" Nov 03 23:05:50.422290 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.422229 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2" Nov 03 23:05:50.465993 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.465957 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:50.465993 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.465993 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:50.466233 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.466010 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wbt\" (UniqueName: \"kubernetes.io/projected/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-kube-api-access-s5wbt\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:50.466233 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.466026 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d07e94e-9f07-4af8-b8fb-1d316bd5705f-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:50.466891 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.466865 4625 scope.go:122] "RemoveContainer" containerID="9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1" Nov 03 23:05:50.467762 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:50.467686 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1\": container with ID starting with 9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1 not found: ID does not exist" containerID="9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1" Nov 03 23:05:50.467879 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.467834 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1"} err="failed to get container status \"9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1\": rpc error: code = NotFound desc = could not find container \"9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1\": container with ID starting with 9d22ff0ae3e87d67f4289ff50567e42fe20cf7a735c3db25756aa5ccaa9178e1 not found: ID does not exist" Nov 03 23:05:50.505061 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.505023 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2"] Nov 03 23:05:50.509315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.509289 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6dcc0-6db687fdf7-gzbq2"] Nov 03 23:05:50.783760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.781587 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d07e94e-9f07-4af8-b8fb-1d316bd5705f" path="/var/lib/kubelet/pods/2d07e94e-9f07-4af8-b8fb-1d316bd5705f/volumes" Nov 03 23:05:50.783760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:50.782412 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3123211a-c11f-4479-be91-2bf9edcb267e" path="/var/lib/kubelet/pods/3123211a-c11f-4479-be91-2bf9edcb267e/volumes" Nov 03 23:05:55.465246 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:55.465213 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j"] Nov 03 23:05:55.465845 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:55.465466 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" containerID="cri-o://63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8" gracePeriod=30 Nov 03 23:05:57.331630 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.331597 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8"] Nov 03 23:05:57.332160 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.331984 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" podUID="431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" containerName="maas-api" containerID="cri-o://f64ddf5641f9bf9e15fae8ec98df61a528a86685ef98dc0bcba2399136a9bd11" gracePeriod=30 Nov 03 23:05:57.507157 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.506956 4625 generic.go:356] "Generic (PLEG): container finished" podID="431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" containerID="f64ddf5641f9bf9e15fae8ec98df61a528a86685ef98dc0bcba2399136a9bd11" exitCode=0 Nov 03 23:05:57.507353 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.507210 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" event={"ID":"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a","Type":"ContainerDied","Data":"f64ddf5641f9bf9e15fae8ec98df61a528a86685ef98dc0bcba2399136a9bd11"} Nov 03 23:05:57.615358 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.615331 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 23:05:57.675915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.675878 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-kube-api-access-m7ndb\") pod \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " Nov 03 23:05:57.676250 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.676231 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-metrics-certs\") pod \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " Nov 03 23:05:57.676458 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.676442 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-openshift-service-ca\") pod \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " Nov 03 23:05:57.679422 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.679398 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-maas-api-tls\") pod \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\" (UID: \"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a\") " Nov 03 23:05:57.681394 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.681364 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-openshift-service-ca" pod "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" (UID: "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:57.684900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.684869 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6"] Nov 03 23:05:57.690850 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.690822 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" podUID="f70a8fa5-192c-486d-825f-d45c9342c6ca" containerName="maas-api" containerID="cri-o://87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c" gracePeriod=30 Nov 03 23:05:57.701184 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.701145 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-maas-api-tls" pod "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" (UID: "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:57.701392 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.701292 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-metrics-certs" pod "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" (UID: "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:57.701957 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.701913 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-kube-api-access-m7ndb" pod "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" (UID: "431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a"). InnerVolumeSpecName "kube-api-access-m7ndb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:57.782379 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.782344 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:57.782555 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.782424 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:57.782555 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.782441 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:57.782555 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.782455 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a-kube-api-access-m7ndb\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:57.949864 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.949838 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 23:05:57.984808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.984781 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70a8fa5-192c-486d-825f-d45c9342c6ca-kube-api-access-snhx7\" (UniqueName: \"kubernetes.io/projected/f70a8fa5-192c-486d-825f-d45c9342c6ca-kube-api-access-snhx7\") pod \"f70a8fa5-192c-486d-825f-d45c9342c6ca\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " Nov 03 23:05:57.984986 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.984815 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70a8fa5-192c-486d-825f-d45c9342c6ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70a8fa5-192c-486d-825f-d45c9342c6ca-openshift-service-ca\") pod \"f70a8fa5-192c-486d-825f-d45c9342c6ca\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " Nov 03 23:05:57.984986 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.984945 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-metrics-certs\") pod \"f70a8fa5-192c-486d-825f-d45c9342c6ca\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " Nov 03 23:05:57.985134 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.985105 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls\") pod \"f70a8fa5-192c-486d-825f-d45c9342c6ca\" (UID: \"f70a8fa5-192c-486d-825f-d45c9342c6ca\") " Nov 03 23:05:57.985278 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.985256 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70a8fa5-192c-486d-825f-d45c9342c6ca-openshift-service-ca" pod "f70a8fa5-192c-486d-825f-d45c9342c6ca" (UID: "f70a8fa5-192c-486d-825f-d45c9342c6ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:57.985591 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.985568 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70a8fa5-192c-486d-825f-d45c9342c6ca-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:57.986998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.986975 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls" pod "f70a8fa5-192c-486d-825f-d45c9342c6ca" (UID: "f70a8fa5-192c-486d-825f-d45c9342c6ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:57.987108 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.987083 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-metrics-certs" pod "f70a8fa5-192c-486d-825f-d45c9342c6ca" (UID: "f70a8fa5-192c-486d-825f-d45c9342c6ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:57.987170 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:57.987131 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70a8fa5-192c-486d-825f-d45c9342c6ca-kube-api-access-snhx7" pod "f70a8fa5-192c-486d-825f-d45c9342c6ca" (UID: "f70a8fa5-192c-486d-825f-d45c9342c6ca"). InnerVolumeSpecName "kube-api-access-snhx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:58.087081 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.087047 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:58.087081 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.087076 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhx7\" (UniqueName: \"kubernetes.io/projected/f70a8fa5-192c-486d-825f-d45c9342c6ca-kube-api-access-snhx7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:58.087081 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.087085 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70a8fa5-192c-486d-825f-d45c9342c6ca-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:05:58.537113 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.535675 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" Nov 03 23:05:58.537113 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.535685 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8" event={"ID":"431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a","Type":"ContainerDied","Data":"32bb7a045b2c754f011557f6eabc55e1b5e216f0b8fb7cff9e1887ec4da7d964"} Nov 03 23:05:58.537113 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.535777 4625 scope.go:122] "RemoveContainer" containerID="f64ddf5641f9bf9e15fae8ec98df61a528a86685ef98dc0bcba2399136a9bd11" Nov 03 23:05:58.540133 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.540023 4625 generic.go:356] "Generic (PLEG): container finished" podID="f70a8fa5-192c-486d-825f-d45c9342c6ca" containerID="87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c" exitCode=0 Nov 03 23:05:58.540264 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.540198 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" Nov 03 23:05:58.540477 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.540446 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" event={"ID":"f70a8fa5-192c-486d-825f-d45c9342c6ca","Type":"ContainerDied","Data":"87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c"} Nov 03 23:05:58.540634 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.540511 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6" event={"ID":"f70a8fa5-192c-486d-825f-d45c9342c6ca","Type":"ContainerDied","Data":"799573b6f8759954f254f1de5197ff08d85367de3037290b020be7e8e23d1e8f"} Nov 03 23:05:58.551954 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.551934 4625 scope.go:122] "RemoveContainer" containerID="87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c" Nov 03 23:05:58.561592 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.561563 4625 scope.go:122] "RemoveContainer" containerID="87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c" Nov 03 23:05:58.561921 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:05:58.561898 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c\": container with ID starting with 87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c not found: ID does not exist" containerID="87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c" Nov 03 23:05:58.562000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.561929 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c"} err="failed to get container status \"87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c\": rpc error: code = NotFound desc = could not find container \"87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c\": container with ID starting with 87a5016014c29c33b02422980a086f656fcff191d65da935bad3058138b5ea6c not found: ID does not exist" Nov 03 23:05:58.600352 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.600320 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8"] Nov 03 23:05:58.612004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.611894 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9b841d-6595b4dd98-tlhg8"] Nov 03 23:05:58.651696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.651661 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6"] Nov 03 23:05:58.657179 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.657150 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-04988c-76667d8677-m8gt6"] Nov 03 23:05:58.729083 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.729052 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" path="/var/lib/kubelet/pods/431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a/volumes" Nov 03 23:05:58.729631 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:58.729614 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70a8fa5-192c-486d-825f-d45c9342c6ca" path="/var/lib/kubelet/pods/f70a8fa5-192c-486d-825f-d45c9342c6ca/volumes" Nov 03 23:05:59.568302 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:59.568270 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r"] Nov 03 23:05:59.568841 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:05:59.568547 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" containerID="cri-o://5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267" gracePeriod=30 Nov 03 23:06:00.747949 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.747886 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 23:06:00.873439 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873408 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-data\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873610 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873449 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-certs\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873610 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873492 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ceb83ca-1c71-40d1-a181-88d931f3538d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb83ca-1c71-40d1-a181-88d931f3538d-istiod-ca-cert\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873610 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873539 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-credential-socket\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873610 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873580 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-token\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873697 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-envoy\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873764 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-data" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:00.873900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873781 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-socket\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873812 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-podinfo\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.873900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.873895 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-kube-api-access-pg22d\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-kube-api-access-pg22d\") pod \"7ceb83ca-1c71-40d1-a181-88d931f3538d\" (UID: \"7ceb83ca-1c71-40d1-a181-88d931f3538d\") " Nov 03 23:06:00.874172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874017 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-certs" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:00.874172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874059 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-socket" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:00.874172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874077 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ceb83ca-1c71-40d1-a181-88d931f3538d-istiod-ca-cert" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:00.874172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874163 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-credential-socket" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:00.874556 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874427 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.874556 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874448 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.874556 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874463 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceb83ca-1c71-40d1-a181-88d931f3538d-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.874556 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874478 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.874556 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.874490 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.876096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.876061 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-podinfo" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:00.876207 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.876161 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-kube-api-access-pg22d" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "kube-api-access-pg22d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:00.876316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.876297 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-envoy" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:00.876316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.876305 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-token" pod "7ceb83ca-1c71-40d1-a181-88d931f3538d" (UID: "7ceb83ca-1c71-40d1-a181-88d931f3538d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:00.975608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.975570 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg22d\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-kube-api-access-pg22d\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.975608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.975598 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.975608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.975607 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.975608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:00.975615 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceb83ca-1c71-40d1-a181-88d931f3538d-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.572271 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.572218 4625 generic.go:356] "Generic (PLEG): container finished" podID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerID="63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8" exitCode=0 Nov 03 23:06:01.572517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.572283 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" event={"ID":"7ceb83ca-1c71-40d1-a181-88d931f3538d","Type":"ContainerDied","Data":"63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8"} Nov 03 23:06:01.572517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.572344 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" Nov 03 23:06:01.572517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.572425 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j" event={"ID":"7ceb83ca-1c71-40d1-a181-88d931f3538d","Type":"ContainerDied","Data":"26648bd369e3a0f4549eb0bf9760040ccadfc7adba569b5a4550adb6eaa39f05"} Nov 03 23:06:01.572517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.572490 4625 scope.go:122] "RemoveContainer" containerID="63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8" Nov 03 23:06:01.584694 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.584668 4625 scope.go:122] "RemoveContainer" containerID="f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094" Nov 03 23:06:01.594963 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.594795 4625 scope.go:122] "RemoveContainer" containerID="63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8" Nov 03 23:06:01.595119 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:01.595096 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8\": container with ID starting with 63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8 not found: ID does not exist" containerID="63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8" Nov 03 23:06:01.595185 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.595131 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8"} err="failed to get container status \"63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8\": rpc error: code = NotFound desc = could not find container \"63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8\": container with ID starting with 63c3a26549cae3fa70ecbfc9b28e29f1202af8b433dd43794b2ff1797edd31f8 not found: ID does not exist" Nov 03 23:06:01.595185 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.595156 4625 scope.go:122] "RemoveContainer" containerID="f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094" Nov 03 23:06:01.595411 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:01.595394 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094\": container with ID starting with f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094 not found: ID does not exist" containerID="f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094" Nov 03 23:06:01.595457 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.595418 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094"} err="failed to get container status \"f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094\": rpc error: code = NotFound desc = could not find container \"f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094\": container with ID starting with f133b414157044bf01571ca79bfb286a0ec39073e62fb6e7d0240540a8a5a094 not found: ID does not exist" Nov 03 23:06:01.636572 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.636540 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j"] Nov 03 23:06:01.648437 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:01.648408 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-7d49j"] Nov 03 23:06:02.187053 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:02.187020 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:06:02.728491 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:02.728461 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" path="/var/lib/kubelet/pods/7ceb83ca-1c71-40d1-a181-88d931f3538d/volumes" Nov 03 23:06:04.873193 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.873163 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:06:04.928573 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928545 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-podinfo\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.928785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928664 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-certs\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.928785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928687 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-socket\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.928785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928708 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-kube-api-access-tvmdc\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-kube-api-access-tvmdc\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.929024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928800 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-data\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.929024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928877 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d2fe03-81da-460a-ba49-df3afe3eefee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d2fe03-81da-460a-ba49-df3afe3eefee-istiod-ca-cert\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.929024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928911 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-credential-socket\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.929024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928942 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-envoy\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.929024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.928994 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-token\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-token\") pod \"26d2fe03-81da-460a-ba49-df3afe3eefee\" (UID: \"26d2fe03-81da-460a-ba49-df3afe3eefee\") " Nov 03 23:06:04.929297 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.929238 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-data" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.929366 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.929333 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-credential-socket" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.929502 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.929459 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-certs" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.929578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.929514 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-socket" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.929578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.929561 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:04.929578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.929575 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:04.929783 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.929624 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d2fe03-81da-460a-ba49-df3afe3eefee-istiod-ca-cert" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:04.931862 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.931738 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-envoy" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.932938 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.932883 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-token" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:04.934254 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.934230 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-kube-api-access-tvmdc" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "kube-api-access-tvmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:04.937889 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:04.937862 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-podinfo" pod "26d2fe03-81da-460a-ba49-df3afe3eefee" (UID: "26d2fe03-81da-460a-ba49-df3afe3eefee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:05.030927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.030885 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d2fe03-81da-460a-ba49-df3afe3eefee-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:05.030927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.030913 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:05.030927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.030922 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:05.030927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.030929 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d2fe03-81da-460a-ba49-df3afe3eefee-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:05.030927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.030937 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:05.030927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.030945 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d2fe03-81da-460a-ba49-df3afe3eefee-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:05.031435 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.030955 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmdc\" (UniqueName: \"kubernetes.io/projected/26d2fe03-81da-460a-ba49-df3afe3eefee-kube-api-access-tvmdc\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:05.610471 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.610434 4625 generic.go:356] "Generic (PLEG): container finished" podID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerID="5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267" exitCode=0 Nov 03 23:06:05.610695 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.610484 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" Nov 03 23:06:05.610695 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.610511 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" event={"ID":"26d2fe03-81da-460a-ba49-df3afe3eefee","Type":"ContainerDied","Data":"5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267"} Nov 03 23:06:05.610695 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.610582 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r" event={"ID":"26d2fe03-81da-460a-ba49-df3afe3eefee","Type":"ContainerDied","Data":"f19539426f2a8a4e5aeda3e4c8fd28b741c9904718f4863f69ee99e8238954ae"} Nov 03 23:06:05.610695 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.610615 4625 scope.go:122] "RemoveContainer" containerID="5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267" Nov 03 23:06:05.622422 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.622402 4625 scope.go:122] "RemoveContainer" containerID="fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876" Nov 03 23:06:05.632033 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.632005 4625 scope.go:122] "RemoveContainer" containerID="5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267" Nov 03 23:06:05.632326 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:05.632305 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267\": container with ID starting with 5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267 not found: ID does not exist" containerID="5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267" Nov 03 23:06:05.632398 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.632335 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267"} err="failed to get container status \"5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267\": rpc error: code = NotFound desc = could not find container \"5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267\": container with ID starting with 5f07bac5d46378286fd168fa959b532dfd0a37bd638885cd7075eec864b4d267 not found: ID does not exist" Nov 03 23:06:05.632398 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.632362 4625 scope.go:122] "RemoveContainer" containerID="fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876" Nov 03 23:06:05.632630 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:05.632610 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876\": container with ID starting with fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876 not found: ID does not exist" containerID="fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876" Nov 03 23:06:05.632684 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.632636 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876"} err="failed to get container status \"fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876\": rpc error: code = NotFound desc = could not find container \"fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876\": container with ID starting with fad7361603110a6296bc04b202bc6479ea07f1ea5da621a73c61eda6ffba4876 not found: ID does not exist" Nov 03 23:06:05.665171 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.665137 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r"] Nov 03 23:06:05.670900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:05.670870 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6dcc0-openshift-default-54f6df97-v8l4r"] Nov 03 23:06:06.150233 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:06.150171 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n"] Nov 03 23:06:06.150833 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:06.150478 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" containerID="cri-o://b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a" gracePeriod=30 Nov 03 23:06:06.778279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:06.778221 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" path="/var/lib/kubelet/pods/26d2fe03-81da-460a-ba49-df3afe3eefee/volumes" Nov 03 23:06:06.832574 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:06.832539 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb"] Nov 03 23:06:06.832862 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:06.832812 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" containerID="cri-o://69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2" gracePeriod=30 Nov 03 23:06:07.339891 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:07.337422 4625 scope.go:122] "RemoveContainer" containerID="f852378ad4498820bea3638803feabd078d3389d1001dafaa0376b89d0909fae" Nov 03 23:06:08.249770 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:08.249707 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:06:08.249950 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:08.249799 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:06:09.232143 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:09.232107 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:06:09.232559 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:09.232161 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:06:10.652590 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:10.652550 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng"] Nov 03 23:06:10.681240 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:10.681203 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm"] Nov 03 23:06:11.502195 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.502169 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 23:06:11.548285 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548251 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-socket\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548499 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548332 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-certs\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548499 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548470 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f93335b-e263-455b-a2eb-d34acc1f1398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f93335b-e263-455b-a2eb-d34acc1f1398-istiod-ca-cert\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548520 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-token\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-token\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548576 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-envoy\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548609 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-socket" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548624 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-data\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548685 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-credential-socket\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548749 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-podinfo\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548755 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-certs" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:11.548803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548786 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-kube-api-access-4lpnq\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-kube-api-access-4lpnq\") pod \"3f93335b-e263-455b-a2eb-d34acc1f1398\" (UID: \"3f93335b-e263-455b-a2eb-d34acc1f1398\") " Nov 03 23:06:11.549339 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.548967 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-credential-socket" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:11.549459 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.549361 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.549459 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.549382 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.549459 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.549395 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.549629 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.549474 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-data" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:11.549629 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.549589 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f93335b-e263-455b-a2eb-d34acc1f1398-istiod-ca-cert" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:11.551937 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.551880 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-token" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:11.552097 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.551778 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-envoy" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:11.552176 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.552099 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-kube-api-access-4lpnq" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "kube-api-access-4lpnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:11.552640 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.552609 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-podinfo" pod "3f93335b-e263-455b-a2eb-d34acc1f1398" (UID: "3f93335b-e263-455b-a2eb-d34acc1f1398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:11.628895 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.628856 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb"] Nov 03 23:06:11.651154 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.651117 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f93335b-e263-455b-a2eb-d34acc1f1398-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.651154 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.651144 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.651154 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.651153 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.651154 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.651167 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.651154 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.651177 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f93335b-e263-455b-a2eb-d34acc1f1398-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.651154 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.651185 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lpnq\" (UniqueName: \"kubernetes.io/projected/3f93335b-e263-455b-a2eb-d34acc1f1398-kube-api-access-4lpnq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.662277 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.662243 4625 generic.go:356] "Generic (PLEG): container finished" podID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerID="b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a" exitCode=0 Nov 03 23:06:11.662797 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.662291 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" Nov 03 23:06:11.662797 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.662321 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" event={"ID":"3f93335b-e263-455b-a2eb-d34acc1f1398","Type":"ContainerDied","Data":"b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a"} Nov 03 23:06:11.662797 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.662389 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n" event={"ID":"3f93335b-e263-455b-a2eb-d34acc1f1398","Type":"ContainerDied","Data":"2e236b4a517cbfbe620c82b6485fc6ab2a14d61a708ddf0502f103225de61ed3"} Nov 03 23:06:11.662797 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.662418 4625 scope.go:122] "RemoveContainer" containerID="b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a" Nov 03 23:06:11.675705 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.675674 4625 scope.go:122] "RemoveContainer" containerID="387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209" Nov 03 23:06:11.688908 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.688880 4625 scope.go:122] "RemoveContainer" containerID="b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a" Nov 03 23:06:11.689274 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:11.689254 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a\": container with ID starting with b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a not found: ID does not exist" containerID="b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a" Nov 03 23:06:11.689328 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.689284 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a"} err="failed to get container status \"b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a\": rpc error: code = NotFound desc = could not find container \"b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a\": container with ID starting with b0f5ccaa0f7c541b52e1443873aa830d99083884df39b0b636ca6959922f524a not found: ID does not exist" Nov 03 23:06:11.689328 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.689302 4625 scope.go:122] "RemoveContainer" containerID="387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209" Nov 03 23:06:11.689597 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:11.689581 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209\": container with ID starting with 387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209 not found: ID does not exist" containerID="387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209" Nov 03 23:06:11.689633 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.689606 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209"} err="failed to get container status \"387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209\": rpc error: code = NotFound desc = could not find container \"387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209\": container with ID starting with 387d5561e426cc2480e0074ec9591969c09e6cf3fb5b31c383b6c896d646f209 not found: ID does not exist" Nov 03 23:06:11.722260 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.722227 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n"] Nov 03 23:06:11.733163 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.733080 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9b841d-openshift-default-797cdf6799-fnr9n"] Nov 03 23:06:11.960674 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.957345 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl"] Nov 03 23:06:11.960674 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:11.957932 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" podUID="2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" containerName="maas-api" containerID="cri-o://0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0" gracePeriod=30 Nov 03 23:06:12.107487 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.107452 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 23:06:12.164539 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.164461 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-podinfo\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.164539 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.164533 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-kube-api-access-rxsnp\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-kube-api-access-rxsnp\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.164826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.164595 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-data\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.164826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.164622 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9903686-a262-41cd-b4fd-2bf5936cddca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9903686-a262-41cd-b4fd-2bf5936cddca-istiod-ca-cert\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.164826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.164698 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-socket\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.164826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.164764 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-envoy\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.165357 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.165197 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-token\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-token\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.165357 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.165286 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-credential-socket\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.165586 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.165399 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-certs\") pod \"f9903686-a262-41cd-b4fd-2bf5936cddca\" (UID: \"f9903686-a262-41cd-b4fd-2bf5936cddca\") " Nov 03 23:06:12.166095 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.165804 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-data" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:12.166095 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.165978 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9903686-a262-41cd-b4fd-2bf5936cddca-istiod-ca-cert" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:12.166429 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.166395 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-credential-socket" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:12.166593 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.166561 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-socket" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:12.166900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.166864 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-certs" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:12.167984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.167961 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-envoy" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:12.168305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.168204 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.168305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.168227 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.168305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.168242 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.168305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.168254 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9903686-a262-41cd-b4fd-2bf5936cddca-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.168305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.168269 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.168305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.168282 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.170485 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.170450 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-podinfo" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:12.171620 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.171586 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-kube-api-access-rxsnp" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "kube-api-access-rxsnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:12.171803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.171781 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-token" pod "f9903686-a262-41cd-b4fd-2bf5936cddca" (UID: "f9903686-a262-41cd-b4fd-2bf5936cddca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:12.272244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.272218 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.272244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.272244 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9903686-a262-41cd-b4fd-2bf5936cddca-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.272388 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.272255 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxsnp\" (UniqueName: \"kubernetes.io/projected/f9903686-a262-41cd-b4fd-2bf5936cddca-kube-api-access-rxsnp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.304636 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.304610 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 23:06:12.373755 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.372728 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-kube-api-access-jlwg9\" (UniqueName: \"kubernetes.io/projected/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-kube-api-access-jlwg9\") pod \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " Nov 03 23:06:12.373755 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.372889 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-maas-api-tls\") pod \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " Nov 03 23:06:12.373755 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.372944 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-openshift-service-ca\") pod \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " Nov 03 23:06:12.373755 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.373111 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-metrics-certs\") pod \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\" (UID: \"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9\") " Nov 03 23:06:12.373755 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.373438 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-openshift-service-ca" pod "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" (UID: "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:12.374161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.373805 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.377165 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.377134 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-maas-api-tls" pod "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" (UID: "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:12.377455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.377432 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-kube-api-access-jlwg9" pod "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" (UID: "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9"). InnerVolumeSpecName "kube-api-access-jlwg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:12.377665 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.377628 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-metrics-certs" pod "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" (UID: "2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:12.476000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.475919 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwg9\" (UniqueName: \"kubernetes.io/projected/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-kube-api-access-jlwg9\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.476000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.475963 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.476000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.475975 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:12.671865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.671836 4625 generic.go:356] "Generic (PLEG): container finished" podID="2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" containerID="0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0" exitCode=0 Nov 03 23:06:12.672346 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.671904 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" Nov 03 23:06:12.672346 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.671928 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" event={"ID":"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9","Type":"ContainerDied","Data":"0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0"} Nov 03 23:06:12.672346 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.671993 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl" event={"ID":"2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9","Type":"ContainerDied","Data":"38f89019a5aa9dd53f7b952c6c4e2cd13f673e5b6d29543a34dc49a562274b5f"} Nov 03 23:06:12.672346 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.672022 4625 scope.go:122] "RemoveContainer" containerID="0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0" Nov 03 23:06:12.685952 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.680904 4625 generic.go:356] "Generic (PLEG): container finished" podID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerID="69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2" exitCode=0 Nov 03 23:06:12.685952 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.681237 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" event={"ID":"f9903686-a262-41cd-b4fd-2bf5936cddca","Type":"ContainerDied","Data":"69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2"} Nov 03 23:06:12.685952 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.681276 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" Nov 03 23:06:12.685952 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.681519 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb" event={"ID":"f9903686-a262-41cd-b4fd-2bf5936cddca","Type":"ContainerDied","Data":"a2b3eea1630fc386aaf27e6bccb31323f58299ad864e4f62591e002ca32ef048"} Nov 03 23:06:12.702526 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.702495 4625 scope.go:122] "RemoveContainer" containerID="0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0" Nov 03 23:06:12.702985 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:12.702954 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0\": container with ID starting with 0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0 not found: ID does not exist" containerID="0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0" Nov 03 23:06:12.703070 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.702995 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0"} err="failed to get container status \"0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0\": rpc error: code = NotFound desc = could not find container \"0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0\": container with ID starting with 0f8abac3f9e87d9201265e7745c6d36d1a5c882e10aa08ae0702b05105c6dcd0 not found: ID does not exist" Nov 03 23:06:12.703070 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.703018 4625 scope.go:122] "RemoveContainer" containerID="69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2" Nov 03 23:06:12.712854 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.712830 4625 scope.go:122] "RemoveContainer" containerID="9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc" Nov 03 23:06:12.737808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.726894 4625 scope.go:122] "RemoveContainer" containerID="69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2" Nov 03 23:06:12.737808 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:12.728821 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2\": container with ID starting with 69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2 not found: ID does not exist" containerID="69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2" Nov 03 23:06:12.737808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.728870 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2"} err="failed to get container status \"69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2\": rpc error: code = NotFound desc = could not find container \"69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2\": container with ID starting with 69d96552465419dfb061f0699080625225e5774c1a9115a3b91aafdffda6c1f2 not found: ID does not exist" Nov 03 23:06:12.737808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.728899 4625 scope.go:122] "RemoveContainer" containerID="9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc" Nov 03 23:06:12.737808 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:12.729426 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc\": container with ID starting with 9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc not found: ID does not exist" containerID="9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc" Nov 03 23:06:12.737808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.729461 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc"} err="failed to get container status \"9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc\": rpc error: code = NotFound desc = could not find container \"9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc\": container with ID starting with 9d7321867043a088afc5d1d871f3082bf2030045b22e2245b8d4eef846b6dafc not found: ID does not exist" Nov 03 23:06:12.737808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.733013 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" path="/var/lib/kubelet/pods/3f93335b-e263-455b-a2eb-d34acc1f1398/volumes" Nov 03 23:06:12.755844 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.755809 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl"] Nov 03 23:06:12.759196 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.759165 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-26f8ba-75bc56676-qwprl"] Nov 03 23:06:12.785838 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.785795 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb"] Nov 03 23:06:12.790085 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:12.790057 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-trdjb"] Nov 03 23:06:14.728696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:14.728661 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" path="/var/lib/kubelet/pods/2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9/volumes" Nov 03 23:06:14.729358 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:14.729336 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" path="/var/lib/kubelet/pods/f9903686-a262-41cd-b4fd-2bf5936cddca/volumes" Nov 03 23:06:15.653815 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:15.653773 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" podUID="ac4ab126-b50f-4673-8ff5-12444e47b637" containerName="payload-processing" containerID="cri-o://40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506" gracePeriod=25 Nov 03 23:06:15.682269 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:15.682215 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" podUID="7fc247d5-97b3-4b2d-97e8-e08346346447" containerName="payload-pre-processing" containerID="cri-o://44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb" gracePeriod=25 Nov 03 23:06:15.955072 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:15.955042 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 23:06:15.975751 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:15.975706 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 23:06:16.128024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.127989 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc247d5-97b3-4b2d-97e8-e08346346447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc247d5-97b3-4b2d-97e8-e08346346447-plugins-config-volume\") pod \"7fc247d5-97b3-4b2d-97e8-e08346346447\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " Nov 03 23:06:16.128219 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.128033 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4ab126-b50f-4673-8ff5-12444e47b637-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4ab126-b50f-4673-8ff5-12444e47b637-plugins-config-volume\") pod \"ac4ab126-b50f-4673-8ff5-12444e47b637\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " Nov 03 23:06:16.128219 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.128081 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4ab126-b50f-4673-8ff5-12444e47b637-kube-api-access-nf7lr\" (UniqueName: \"kubernetes.io/projected/ac4ab126-b50f-4673-8ff5-12444e47b637-kube-api-access-nf7lr\") pod \"ac4ab126-b50f-4673-8ff5-12444e47b637\" (UID: \"ac4ab126-b50f-4673-8ff5-12444e47b637\") " Nov 03 23:06:16.128219 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.128178 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc247d5-97b3-4b2d-97e8-e08346346447-kube-api-access-9ghgq\" (UniqueName: \"kubernetes.io/projected/7fc247d5-97b3-4b2d-97e8-e08346346447-kube-api-access-9ghgq\") pod \"7fc247d5-97b3-4b2d-97e8-e08346346447\" (UID: \"7fc247d5-97b3-4b2d-97e8-e08346346447\") " Nov 03 23:06:16.128449 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.128426 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc247d5-97b3-4b2d-97e8-e08346346447-plugins-config-volume" pod "7fc247d5-97b3-4b2d-97e8-e08346346447" (UID: "7fc247d5-97b3-4b2d-97e8-e08346346447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:16.128525 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.128503 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4ab126-b50f-4673-8ff5-12444e47b637-plugins-config-volume" pod "ac4ab126-b50f-4673-8ff5-12444e47b637" (UID: "ac4ab126-b50f-4673-8ff5-12444e47b637"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:16.128645 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.128623 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc247d5-97b3-4b2d-97e8-e08346346447-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:16.128645 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.128641 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4ab126-b50f-4673-8ff5-12444e47b637-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:16.130294 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.130277 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4ab126-b50f-4673-8ff5-12444e47b637-kube-api-access-nf7lr" pod "ac4ab126-b50f-4673-8ff5-12444e47b637" (UID: "ac4ab126-b50f-4673-8ff5-12444e47b637"). InnerVolumeSpecName "kube-api-access-nf7lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:16.130404 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.130382 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc247d5-97b3-4b2d-97e8-e08346346447-kube-api-access-9ghgq" pod "7fc247d5-97b3-4b2d-97e8-e08346346447" (UID: "7fc247d5-97b3-4b2d-97e8-e08346346447"). InnerVolumeSpecName "kube-api-access-9ghgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:16.230317 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.230233 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf7lr\" (UniqueName: \"kubernetes.io/projected/ac4ab126-b50f-4673-8ff5-12444e47b637-kube-api-access-nf7lr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:16.230317 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.230264 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ghgq\" (UniqueName: \"kubernetes.io/projected/7fc247d5-97b3-4b2d-97e8-e08346346447-kube-api-access-9ghgq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:16.715789 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.715751 4625 generic.go:356] "Generic (PLEG): container finished" podID="ac4ab126-b50f-4673-8ff5-12444e47b637" containerID="40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506" exitCode=0 Nov 03 23:06:16.715947 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.715802 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" Nov 03 23:06:16.715947 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.715804 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" event={"ID":"ac4ab126-b50f-4673-8ff5-12444e47b637","Type":"ContainerDied","Data":"40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506"} Nov 03 23:06:16.715947 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.715868 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng" event={"ID":"ac4ab126-b50f-4673-8ff5-12444e47b637","Type":"ContainerDied","Data":"14d92705f74706016a4a63baa46a177739a48818617682849acb7cd49bd8fcbc"} Nov 03 23:06:16.715947 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.715900 4625 scope.go:122] "RemoveContainer" containerID="40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506" Nov 03 23:06:16.717384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.717358 4625 generic.go:356] "Generic (PLEG): container finished" podID="7fc247d5-97b3-4b2d-97e8-e08346346447" containerID="44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb" exitCode=0 Nov 03 23:06:16.717500 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.717398 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" Nov 03 23:06:16.717500 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.717419 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" event={"ID":"7fc247d5-97b3-4b2d-97e8-e08346346447","Type":"ContainerDied","Data":"44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb"} Nov 03 23:06:16.717500 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.717465 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm" event={"ID":"7fc247d5-97b3-4b2d-97e8-e08346346447","Type":"ContainerDied","Data":"150bca91491b8028fc7ff6b3ef12dfacc0d46355ed48e36fc79eefcd214d8d15"} Nov 03 23:06:16.726383 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.726276 4625 scope.go:122] "RemoveContainer" containerID="40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506" Nov 03 23:06:16.726581 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:16.726556 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506\": container with ID starting with 40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506 not found: ID does not exist" containerID="40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506" Nov 03 23:06:16.726657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.726589 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506"} err="failed to get container status \"40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506\": rpc error: code = NotFound desc = could not find container \"40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506\": container with ID starting with 40e3fb11fcac91f82578d619332390dea2fde3579c258550d54561a5c57f3506 not found: ID does not exist" Nov 03 23:06:16.726657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.726611 4625 scope.go:122] "RemoveContainer" containerID="44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb" Nov 03 23:06:16.737435 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.737410 4625 scope.go:122] "RemoveContainer" containerID="44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb" Nov 03 23:06:16.737763 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:16.737740 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb\": container with ID starting with 44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb not found: ID does not exist" containerID="44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb" Nov 03 23:06:16.737869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.737769 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb"} err="failed to get container status \"44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb\": rpc error: code = NotFound desc = could not find container \"44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb\": container with ID starting with 44971898043f6e3d166fc9372c091d5244e3ab1de4457a56f589b5933be6c5eb not found: ID does not exist" Nov 03 23:06:16.770693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.770656 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm"] Nov 03 23:06:16.782270 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.782238 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-0847aa-6d5b985b7-xt8mm"] Nov 03 23:06:16.817398 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.817355 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng"] Nov 03 23:06:16.821845 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:16.821780 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0847aa-fc76d97d-m54ng"] Nov 03 23:06:17.567126 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.567089 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2"] Nov 03 23:06:17.567655 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.567341 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" podUID="7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" containerName="maas-api" containerID="cri-o://b8f44f995d672a7e100ca5c49dd7e340fdb0ca622636c973c336e453e03c5d41" gracePeriod=30 Nov 03 23:06:17.729292 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.729258 4625 generic.go:356] "Generic (PLEG): container finished" podID="7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" containerID="b8f44f995d672a7e100ca5c49dd7e340fdb0ca622636c973c336e453e03c5d41" exitCode=0 Nov 03 23:06:17.729466 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.729335 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" event={"ID":"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3","Type":"ContainerDied","Data":"b8f44f995d672a7e100ca5c49dd7e340fdb0ca622636c973c336e453e03c5d41"} Nov 03 23:06:17.847992 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.847961 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 23:06:17.962138 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.962085 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-openshift-service-ca\") pod \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " Nov 03 23:06:17.962406 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.962156 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-maas-api-tls\") pod \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " Nov 03 23:06:17.962406 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.962220 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-metrics-certs\") pod \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " Nov 03 23:06:17.962406 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.962279 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-kube-api-access-mb2sk\" (UniqueName: \"kubernetes.io/projected/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-kube-api-access-mb2sk\") pod \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\" (UID: \"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3\") " Nov 03 23:06:17.962788 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.962566 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-openshift-service-ca" pod "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" (UID: "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:17.962882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.962852 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:17.964945 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.964803 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-kube-api-access-mb2sk" pod "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" (UID: "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3"). InnerVolumeSpecName "kube-api-access-mb2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:17.965090 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.964964 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-maas-api-tls" pod "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" (UID: "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:17.965090 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:17.964982 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-metrics-certs" pod "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" (UID: "7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:18.064115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.064071 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:18.064115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.064111 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:18.064355 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.064126 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2sk\" (UniqueName: \"kubernetes.io/projected/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3-kube-api-access-mb2sk\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:18.727314 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.727275 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc247d5-97b3-4b2d-97e8-e08346346447" path="/var/lib/kubelet/pods/7fc247d5-97b3-4b2d-97e8-e08346346447/volumes" Nov 03 23:06:18.727914 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.727894 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4ab126-b50f-4673-8ff5-12444e47b637" path="/var/lib/kubelet/pods/ac4ab126-b50f-4673-8ff5-12444e47b637/volumes" Nov 03 23:06:18.741036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.737541 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" event={"ID":"7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3","Type":"ContainerDied","Data":"b3a9e211e68c7662639da0fd2ac092c03628273bdab585f5f079a8e9c6832147"} Nov 03 23:06:18.741036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.737600 4625 scope.go:122] "RemoveContainer" containerID="b8f44f995d672a7e100ca5c49dd7e340fdb0ca622636c973c336e453e03c5d41" Nov 03 23:06:18.741036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.737543 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2" Nov 03 23:06:18.790375 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.790334 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2"] Nov 03 23:06:18.794162 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:18.794129 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0847aa-757ff4cccf-nvzn2"] Nov 03 23:06:20.727435 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:20.727402 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" path="/var/lib/kubelet/pods/7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3/volumes" Nov 03 23:06:25.248742 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:25.248689 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj"] Nov 03 23:06:25.249360 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:25.248957 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" containerID="cri-o://ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c" gracePeriod=30 Nov 03 23:06:26.093351 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:26.093318 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s"] Nov 03 23:06:26.650739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:26.650677 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" containerID="cri-o://5135fe300efd3fee97b61c94442a56afd296d08a3587889dacee5dff582ea80b" gracePeriod=45 Nov 03 23:06:26.804215 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:26.804184 4625 generic.go:356] "Generic (PLEG): container finished" podID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerID="5135fe300efd3fee97b61c94442a56afd296d08a3587889dacee5dff582ea80b" exitCode=0 Nov 03 23:06:26.804424 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:26.804258 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" event={"ID":"9de0a5b4-30e8-440b-8a10-832f72c5e885","Type":"ContainerDied","Data":"5135fe300efd3fee97b61c94442a56afd296d08a3587889dacee5dff582ea80b"} Nov 03 23:06:26.915911 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:26.915850 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 23:06:27.003388 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.003349 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-dshm\") pod \"9de0a5b4-30e8-440b-8a10-832f72c5e885\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " Nov 03 23:06:27.003558 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.003493 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de0a5b4-30e8-440b-8a10-832f72c5e885-tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0a5b4-30e8-440b-8a10-832f72c5e885-tls-certs\") pod \"9de0a5b4-30e8-440b-8a10-832f72c5e885\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " Nov 03 23:06:27.003607 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.003562 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-home\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-home\") pod \"9de0a5b4-30e8-440b-8a10-832f72c5e885\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " Nov 03 23:06:27.003669 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.003613 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de0a5b4-30e8-440b-8a10-832f72c5e885-kube-api-access-fkb26\" (UniqueName: \"kubernetes.io/projected/9de0a5b4-30e8-440b-8a10-832f72c5e885-kube-api-access-fkb26\") pod \"9de0a5b4-30e8-440b-8a10-832f72c5e885\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " Nov 03 23:06:27.003752 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.003665 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-tmp-dir\") pod \"9de0a5b4-30e8-440b-8a10-832f72c5e885\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " Nov 03 23:06:27.003752 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.003705 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-model-cache\") pod \"9de0a5b4-30e8-440b-8a10-832f72c5e885\" (UID: \"9de0a5b4-30e8-440b-8a10-832f72c5e885\") " Nov 03 23:06:27.003975 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.003940 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-home" pod "9de0a5b4-30e8-440b-8a10-832f72c5e885" (UID: "9de0a5b4-30e8-440b-8a10-832f72c5e885"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.004134 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.004108 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-tmp-dir" pod "9de0a5b4-30e8-440b-8a10-832f72c5e885" (UID: "9de0a5b4-30e8-440b-8a10-832f72c5e885"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.004216 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.004194 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-model-cache" pod "9de0a5b4-30e8-440b-8a10-832f72c5e885" (UID: "9de0a5b4-30e8-440b-8a10-832f72c5e885"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.004343 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.004327 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.004425 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.004348 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.004425 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.004362 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.005644 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.005619 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-dshm" pod "9de0a5b4-30e8-440b-8a10-832f72c5e885" (UID: "9de0a5b4-30e8-440b-8a10-832f72c5e885"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.006203 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.006171 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de0a5b4-30e8-440b-8a10-832f72c5e885-tls-certs" pod "9de0a5b4-30e8-440b-8a10-832f72c5e885" (UID: "9de0a5b4-30e8-440b-8a10-832f72c5e885"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:27.006320 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.006239 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de0a5b4-30e8-440b-8a10-832f72c5e885-kube-api-access-fkb26" pod "9de0a5b4-30e8-440b-8a10-832f72c5e885" (UID: "9de0a5b4-30e8-440b-8a10-832f72c5e885"). InnerVolumeSpecName "kube-api-access-fkb26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:27.105128 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.105098 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de0a5b4-30e8-440b-8a10-832f72c5e885-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.105128 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.105123 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0a5b4-30e8-440b-8a10-832f72c5e885-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.105128 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.105134 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkb26\" (UniqueName: \"kubernetes.io/projected/9de0a5b4-30e8-440b-8a10-832f72c5e885-kube-api-access-fkb26\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.811632 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.811606 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" Nov 03 23:06:27.811632 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.811628 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb" event={"ID":"9de0a5b4-30e8-440b-8a10-832f72c5e885","Type":"ContainerDied","Data":"7e16f5c73c1beecb5ca3f6aa4323b5b3a74fd3ea9a2435cbd082be69b799c4cf"} Nov 03 23:06:27.812216 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.811673 4625 scope.go:122] "RemoveContainer" containerID="5135fe300efd3fee97b61c94442a56afd296d08a3587889dacee5dff582ea80b" Nov 03 23:06:27.891706 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.891658 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb"] Nov 03 23:06:27.895003 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:27.894975 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-2-w5-434ce3-kserve-5c5d69c5f5-klndb"] Nov 03 23:06:28.734802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:28.734762 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" path="/var/lib/kubelet/pods/9de0a5b4-30e8-440b-8a10-832f72c5e885/volumes" Nov 03 23:06:30.519607 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.519548 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:06:30.551515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551478 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89dabd04-7e6e-46cd-9331-ced0975c71ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dabd04-7e6e-46cd-9331-ced0975c71ca-istiod-ca-cert\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.551738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551530 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-certs\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.551738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551554 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-credential-socket\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.551738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551611 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-socket\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.551738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551651 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-data\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.551738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551692 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-token\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-token\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.552036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551763 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-envoy\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.552036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551853 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-podinfo\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.552036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551895 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-certs" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:30.552036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551917 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-kube-api-access-2g7vt\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-kube-api-access-2g7vt\") pod \"89dabd04-7e6e-46cd-9331-ced0975c71ca\" (UID: \"89dabd04-7e6e-46cd-9331-ced0975c71ca\") " Nov 03 23:06:30.552036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.551958 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dabd04-7e6e-46cd-9331-ced0975c71ca-istiod-ca-cert" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:30.552288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.552249 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-data" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:30.552423 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.552402 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89dabd04-7e6e-46cd-9331-ced0975c71ca-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.552512 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.552456 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.552512 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.552469 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.552512 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.552493 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-socket" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:30.552807 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.552780 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-credential-socket" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:30.555194 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.555158 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-token" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:30.555685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.555660 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-podinfo" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:30.556581 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.556126 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-kube-api-access-2g7vt" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "kube-api-access-2g7vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:30.557758 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.557061 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-envoy" pod "89dabd04-7e6e-46cd-9331-ced0975c71ca" (UID: "89dabd04-7e6e-46cd-9331-ced0975c71ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:30.653559 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.653516 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.653559 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.653556 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.653836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.653569 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89dabd04-7e6e-46cd-9331-ced0975c71ca-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.653836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.653583 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7vt\" (UniqueName: \"kubernetes.io/projected/89dabd04-7e6e-46cd-9331-ced0975c71ca-kube-api-access-2g7vt\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.653836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.653597 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.653836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.653612 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89dabd04-7e6e-46cd-9331-ced0975c71ca-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.835029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.834937 4625 generic.go:356] "Generic (PLEG): container finished" podID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerID="ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c" exitCode=0 Nov 03 23:06:30.835029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.834980 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" event={"ID":"89dabd04-7e6e-46cd-9331-ced0975c71ca","Type":"ContainerDied","Data":"ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c"} Nov 03 23:06:30.835029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.834990 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" Nov 03 23:06:30.835313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.835032 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj" event={"ID":"89dabd04-7e6e-46cd-9331-ced0975c71ca","Type":"ContainerDied","Data":"5dccd62076e0b51f88e838b49e94103f454218f58475af7edc2d09232d17a7d6"} Nov 03 23:06:30.835313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.835063 4625 scope.go:122] "RemoveContainer" containerID="ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c" Nov 03 23:06:30.845693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.845664 4625 scope.go:122] "RemoveContainer" containerID="ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c" Nov 03 23:06:30.846031 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:30.846007 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c\": container with ID starting with ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c not found: ID does not exist" containerID="ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c" Nov 03 23:06:30.846131 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.846043 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c"} err="failed to get container status \"ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c\": rpc error: code = NotFound desc = could not find container \"ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c\": container with ID starting with ee42ddd57251dec7e141805a301a84191abcffc8f85a5bd446fb20151ca0953c not found: ID does not exist" Nov 03 23:06:30.887210 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.887165 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj"] Nov 03 23:06:30.900377 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:30.900337 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-pt6sj"] Nov 03 23:06:32.187838 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:32.187805 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:06:32.747742 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:32.747151 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" path="/var/lib/kubelet/pods/89dabd04-7e6e-46cd-9331-ced0975c71ca/volumes" Nov 03 23:06:40.156434 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.156399 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj"] Nov 03 23:06:40.156915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.156624 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" podUID="1c53996e-be1a-4df1-a7c7-4bd4daecda71" containerName="maas-api" containerID="cri-o://12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add" gracePeriod=30 Nov 03 23:06:40.701771 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.701741 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 23:06:40.805873 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.805840 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-metrics-certs\") pod \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " Nov 03 23:06:40.806109 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.805909 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-maas-api-tls\") pod \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " Nov 03 23:06:40.806109 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.806024 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c53996e-be1a-4df1-a7c7-4bd4daecda71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c53996e-be1a-4df1-a7c7-4bd4daecda71-openshift-service-ca\") pod \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " Nov 03 23:06:40.806109 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.806043 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c53996e-be1a-4df1-a7c7-4bd4daecda71-kube-api-access-2hzdw\" (UniqueName: \"kubernetes.io/projected/1c53996e-be1a-4df1-a7c7-4bd4daecda71-kube-api-access-2hzdw\") pod \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\" (UID: \"1c53996e-be1a-4df1-a7c7-4bd4daecda71\") " Nov 03 23:06:40.806430 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.806399 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c53996e-be1a-4df1-a7c7-4bd4daecda71-openshift-service-ca" pod "1c53996e-be1a-4df1-a7c7-4bd4daecda71" (UID: "1c53996e-be1a-4df1-a7c7-4bd4daecda71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:40.808143 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.808117 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-maas-api-tls" pod "1c53996e-be1a-4df1-a7c7-4bd4daecda71" (UID: "1c53996e-be1a-4df1-a7c7-4bd4daecda71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:40.808523 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.808506 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-metrics-certs" pod "1c53996e-be1a-4df1-a7c7-4bd4daecda71" (UID: "1c53996e-be1a-4df1-a7c7-4bd4daecda71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:40.808608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.808565 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c53996e-be1a-4df1-a7c7-4bd4daecda71-kube-api-access-2hzdw" pod "1c53996e-be1a-4df1-a7c7-4bd4daecda71" (UID: "1c53996e-be1a-4df1-a7c7-4bd4daecda71"). InnerVolumeSpecName "kube-api-access-2hzdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:40.908865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.908070 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c53996e-be1a-4df1-a7c7-4bd4daecda71-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:40.908865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.908115 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hzdw\" (UniqueName: \"kubernetes.io/projected/1c53996e-be1a-4df1-a7c7-4bd4daecda71-kube-api-access-2hzdw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:40.908865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.908133 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:40.908865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.908145 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c53996e-be1a-4df1-a7c7-4bd4daecda71-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:40.952819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.944626 4625 generic.go:356] "Generic (PLEG): container finished" podID="1c53996e-be1a-4df1-a7c7-4bd4daecda71" containerID="12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add" exitCode=0 Nov 03 23:06:40.952819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.944678 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" Nov 03 23:06:40.952819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.944800 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" event={"ID":"1c53996e-be1a-4df1-a7c7-4bd4daecda71","Type":"ContainerDied","Data":"12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add"} Nov 03 23:06:40.952819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.944851 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj" event={"ID":"1c53996e-be1a-4df1-a7c7-4bd4daecda71","Type":"ContainerDied","Data":"a116922ae616f2034f36a0ac04e8e3d33c90af410f30afc602c110d46a561a0b"} Nov 03 23:06:40.952819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.944884 4625 scope.go:122] "RemoveContainer" containerID="12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add" Nov 03 23:06:40.959909 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.959882 4625 scope.go:122] "RemoveContainer" containerID="12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add" Nov 03 23:06:40.960252 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:40.960228 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add\": container with ID starting with 12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add not found: ID does not exist" containerID="12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add" Nov 03 23:06:40.960352 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:40.960266 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add"} err="failed to get container status \"12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add\": rpc error: code = NotFound desc = could not find container \"12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add\": container with ID starting with 12a46f64b99206f8f9c96a3a3df5663c94aef3b93b62aee11ee62112cd646add not found: ID does not exist" Nov 03 23:06:41.039063 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.039015 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj"] Nov 03 23:06:41.061811 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.061768 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-23d281-7d7d88cb89-smvtj"] Nov 03 23:06:41.115595 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.115522 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" containerID="cri-o://60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1" gracePeriod=45 Nov 03 23:06:41.267803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.267167 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw"] Nov 03 23:06:41.267803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.267415 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" containerID="cri-o://fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7" gracePeriod=30 Nov 03 23:06:41.556743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.550499 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 23:06:41.627270 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.627234 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-model-cache\") pod \"b30970e3-1702-45fa-bc60-bcd754afa716\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " Nov 03 23:06:41.627446 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.627292 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-tmp-dir\") pod \"b30970e3-1702-45fa-bc60-bcd754afa716\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " Nov 03 23:06:41.627604 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.627572 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-model-cache" pod "b30970e3-1702-45fa-bc60-bcd754afa716" (UID: "b30970e3-1702-45fa-bc60-bcd754afa716"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.627604 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.627588 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-tmp-dir" pod "b30970e3-1702-45fa-bc60-bcd754afa716" (UID: "b30970e3-1702-45fa-bc60-bcd754afa716"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.627865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.627843 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.627939 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.627876 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.728480 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.728441 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-home\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-home\") pod \"b30970e3-1702-45fa-bc60-bcd754afa716\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " Nov 03 23:06:41.728480 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.728485 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-dshm\") pod \"b30970e3-1702-45fa-bc60-bcd754afa716\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " Nov 03 23:06:41.728772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.728529 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30970e3-1702-45fa-bc60-bcd754afa716-tls-certs\" (UniqueName: \"kubernetes.io/secret/b30970e3-1702-45fa-bc60-bcd754afa716-tls-certs\") pod \"b30970e3-1702-45fa-bc60-bcd754afa716\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " Nov 03 23:06:41.728772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.728556 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30970e3-1702-45fa-bc60-bcd754afa716-kube-api-access-2gg7c\" (UniqueName: \"kubernetes.io/projected/b30970e3-1702-45fa-bc60-bcd754afa716-kube-api-access-2gg7c\") pod \"b30970e3-1702-45fa-bc60-bcd754afa716\" (UID: \"b30970e3-1702-45fa-bc60-bcd754afa716\") " Nov 03 23:06:41.729011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.728796 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-home" pod "b30970e3-1702-45fa-bc60-bcd754afa716" (UID: "b30970e3-1702-45fa-bc60-bcd754afa716"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.729079 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.729033 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.731169 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.731138 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30970e3-1702-45fa-bc60-bcd754afa716-tls-certs" pod "b30970e3-1702-45fa-bc60-bcd754afa716" (UID: "b30970e3-1702-45fa-bc60-bcd754afa716"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:41.731283 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.731201 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-dshm" pod "b30970e3-1702-45fa-bc60-bcd754afa716" (UID: "b30970e3-1702-45fa-bc60-bcd754afa716"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.731347 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.731323 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30970e3-1702-45fa-bc60-bcd754afa716-kube-api-access-2gg7c" pod "b30970e3-1702-45fa-bc60-bcd754afa716" (UID: "b30970e3-1702-45fa-bc60-bcd754afa716"). InnerVolumeSpecName "kube-api-access-2gg7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:41.830316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.830222 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30970e3-1702-45fa-bc60-bcd754afa716-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.830316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.830260 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30970e3-1702-45fa-bc60-bcd754afa716-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.830316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.830273 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gg7c\" (UniqueName: \"kubernetes.io/projected/b30970e3-1702-45fa-bc60-bcd754afa716-kube-api-access-2gg7c\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.953513 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.953473 4625 generic.go:356] "Generic (PLEG): container finished" podID="b30970e3-1702-45fa-bc60-bcd754afa716" containerID="60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1" exitCode=0 Nov 03 23:06:41.953745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.953517 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" Nov 03 23:06:41.953745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.953536 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" event={"ID":"b30970e3-1702-45fa-bc60-bcd754afa716","Type":"ContainerDied","Data":"60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1"} Nov 03 23:06:41.953745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.953567 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" event={"ID":"b30970e3-1702-45fa-bc60-bcd754afa716","Type":"ContainerDied","Data":"da3e559cf3ceae17cfbc2d2a9ef99299b5b0d43113014ad19c438e584cbb2c03"} Nov 03 23:06:41.953745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.953583 4625 scope.go:122] "RemoveContainer" containerID="60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1" Nov 03 23:06:41.965413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.965390 4625 scope.go:122] "RemoveContainer" containerID="60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1" Nov 03 23:06:41.965676 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:41.965659 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1\": container with ID starting with 60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1 not found: ID does not exist" containerID="60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1" Nov 03 23:06:41.965752 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:41.965684 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1"} err="failed to get container status \"60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1\": rpc error: code = NotFound desc = could not find container \"60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1\": container with ID starting with 60324a1c961ccf560640b291bbd2457f8fbd2dd3e73dbce52096864ca2df25b1 not found: ID does not exist" Nov 03 23:06:42.003936 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:42.003892 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s"] Nov 03 23:06:42.007680 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:42.007654 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s"] Nov 03 23:06:42.440491 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:42.440450 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-2-w6-2d619f-kserve-6cf4d8ffc6-fqw4s" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:06:42.727127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:42.727046 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c53996e-be1a-4df1-a7c7-4bd4daecda71" path="/var/lib/kubelet/pods/1c53996e-be1a-4df1-a7c7-4bd4daecda71/volumes" Nov 03 23:06:42.727517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:42.727501 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" path="/var/lib/kubelet/pods/b30970e3-1702-45fa-bc60-bcd754afa716/volumes" Nov 03 23:06:44.017089 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.017030 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7"] Nov 03 23:06:44.017944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.017890 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" podUID="676907ab-2cf3-4909-810b-848642fe6bb7" containerName="maas-api" containerID="cri-o://6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74" gracePeriod=30 Nov 03 23:06:44.149074 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.149036 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf"] Nov 03 23:06:44.149359 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.149306 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" containerID="cri-o://dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba" gracePeriod=30 Nov 03 23:06:44.289557 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.289530 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 23:06:44.361070 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.361033 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls\") pod \"676907ab-2cf3-4909-810b-848642fe6bb7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " Nov 03 23:06:44.361282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.361101 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676907ab-2cf3-4909-810b-848642fe6bb7-kube-api-access-7cxjw\" (UniqueName: \"kubernetes.io/projected/676907ab-2cf3-4909-810b-848642fe6bb7-kube-api-access-7cxjw\") pod \"676907ab-2cf3-4909-810b-848642fe6bb7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " Nov 03 23:06:44.361282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.361197 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-metrics-certs\") pod \"676907ab-2cf3-4909-810b-848642fe6bb7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " Nov 03 23:06:44.361282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.361239 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676907ab-2cf3-4909-810b-848642fe6bb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676907ab-2cf3-4909-810b-848642fe6bb7-openshift-service-ca\") pod \"676907ab-2cf3-4909-810b-848642fe6bb7\" (UID: \"676907ab-2cf3-4909-810b-848642fe6bb7\") " Nov 03 23:06:44.361746 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.361691 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676907ab-2cf3-4909-810b-848642fe6bb7-openshift-service-ca" pod "676907ab-2cf3-4909-810b-848642fe6bb7" (UID: "676907ab-2cf3-4909-810b-848642fe6bb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:44.363749 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.363677 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls" pod "676907ab-2cf3-4909-810b-848642fe6bb7" (UID: "676907ab-2cf3-4909-810b-848642fe6bb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:44.363940 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.363911 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676907ab-2cf3-4909-810b-848642fe6bb7-kube-api-access-7cxjw" pod "676907ab-2cf3-4909-810b-848642fe6bb7" (UID: "676907ab-2cf3-4909-810b-848642fe6bb7"). InnerVolumeSpecName "kube-api-access-7cxjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:44.364811 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.364777 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-metrics-certs" pod "676907ab-2cf3-4909-810b-848642fe6bb7" (UID: "676907ab-2cf3-4909-810b-848642fe6bb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:44.463027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.462984 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:44.463027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.463019 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxjw\" (UniqueName: \"kubernetes.io/projected/676907ab-2cf3-4909-810b-848642fe6bb7-kube-api-access-7cxjw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:44.463027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.463031 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676907ab-2cf3-4909-810b-848642fe6bb7-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:44.463279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.463041 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676907ab-2cf3-4909-810b-848642fe6bb7-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:44.975597 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.975565 4625 generic.go:356] "Generic (PLEG): container finished" podID="676907ab-2cf3-4909-810b-848642fe6bb7" containerID="6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74" exitCode=0 Nov 03 23:06:44.975775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.975615 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" Nov 03 23:06:44.975775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.975652 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" event={"ID":"676907ab-2cf3-4909-810b-848642fe6bb7","Type":"ContainerDied","Data":"6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74"} Nov 03 23:06:44.975775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.975711 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7" event={"ID":"676907ab-2cf3-4909-810b-848642fe6bb7","Type":"ContainerDied","Data":"684f2b32f4622bda3929db2e86fcf4843fced7185736ea1e703a98f6efcadf22"} Nov 03 23:06:44.975775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.975754 4625 scope.go:122] "RemoveContainer" containerID="6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74" Nov 03 23:06:44.985619 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.985593 4625 scope.go:122] "RemoveContainer" containerID="6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74" Nov 03 23:06:44.985921 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:44.985902 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74\": container with ID starting with 6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74 not found: ID does not exist" containerID="6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74" Nov 03 23:06:44.985994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:44.985928 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74"} err="failed to get container status \"6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74\": rpc error: code = NotFound desc = could not find container \"6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74\": container with ID starting with 6cead64d40cdabfbcb8e0481d16c7f56848e775f5505d78ee0a6e5aecfd10a74 not found: ID does not exist" Nov 03 23:06:45.031081 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:45.031055 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7"] Nov 03 23:06:45.042564 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:45.042536 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a940d5-755548d596-m8lc7"] Nov 03 23:06:46.656197 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.656166 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 23:06:46.684513 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684478 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-certs\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684710 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684534 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-istio-token\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-istio-token\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684710 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684577 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-credential-socket\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684710 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684633 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-socket\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684710 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684676 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6725a80-3ec7-436a-a031-240f2c83e623-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6725a80-3ec7-436a-a031-240f2c83e623-istio-podinfo\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684746 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-kube-api-access-wbtxq\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-kube-api-access-wbtxq\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684799 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6725a80-3ec7-436a-a031-240f2c83e623-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6725a80-3ec7-436a-a031-240f2c83e623-istiod-ca-cert\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684851 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-data\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.684981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684854 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-certs" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:46.684981 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.684899 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-envoy\") pod \"e6725a80-3ec7-436a-a031-240f2c83e623\" (UID: \"e6725a80-3ec7-436a-a031-240f2c83e623\") " Nov 03 23:06:46.685316 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.685277 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.685376 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.685334 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-credential-socket" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:46.685455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.685409 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-socket" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:46.685526 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.685497 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-data" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:46.685853 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.685828 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6725a80-3ec7-436a-a031-240f2c83e623-istiod-ca-cert" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:46.687259 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.687234 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-istio-token" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:46.687615 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.687586 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-envoy" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:46.687883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.687866 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-kube-api-access-wbtxq" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "kube-api-access-wbtxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:46.687953 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.687899 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6725a80-3ec7-436a-a031-240f2c83e623-istio-podinfo" pod "e6725a80-3ec7-436a-a031-240f2c83e623" (UID: "e6725a80-3ec7-436a-a031-240f2c83e623"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:46.726982 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.726909 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676907ab-2cf3-4909-810b-848642fe6bb7" path="/var/lib/kubelet/pods/676907ab-2cf3-4909-810b-848642fe6bb7/volumes" Nov 03 23:06:46.786371 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786332 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.786371 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786368 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.786584 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786383 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.786584 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786398 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.786584 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786412 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6725a80-3ec7-436a-a031-240f2c83e623-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.786584 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786428 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6725a80-3ec7-436a-a031-240f2c83e623-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.786584 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786443 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbtxq\" (UniqueName: \"kubernetes.io/projected/e6725a80-3ec7-436a-a031-240f2c83e623-kube-api-access-wbtxq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.786584 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.786457 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6725a80-3ec7-436a-a031-240f2c83e623-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:46.991937 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.991847 4625 generic.go:356] "Generic (PLEG): container finished" podID="e6725a80-3ec7-436a-a031-240f2c83e623" containerID="fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7" exitCode=0 Nov 03 23:06:46.991937 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.991905 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" Nov 03 23:06:46.992194 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.991943 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" event={"ID":"e6725a80-3ec7-436a-a031-240f2c83e623","Type":"ContainerDied","Data":"fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7"} Nov 03 23:06:46.992194 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.992008 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw" event={"ID":"e6725a80-3ec7-436a-a031-240f2c83e623","Type":"ContainerDied","Data":"e4786107802de5137398e851fc83db2c6a81ae80e7d97b3c5f0e2ec76c7c9a66"} Nov 03 23:06:46.992194 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:46.992041 4625 scope.go:122] "RemoveContainer" containerID="fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7" Nov 03 23:06:47.012780 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:47.012757 4625 scope.go:122] "RemoveContainer" containerID="2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80" Nov 03 23:06:47.021909 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:47.021886 4625 scope.go:122] "RemoveContainer" containerID="fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7" Nov 03 23:06:47.022193 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:47.022175 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7\": container with ID starting with fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7 not found: ID does not exist" containerID="fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7" Nov 03 23:06:47.022286 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:47.022208 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7"} err="failed to get container status \"fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7\": rpc error: code = NotFound desc = could not find container \"fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7\": container with ID starting with fc161c8c32ac6c1623f7708f2ee77b883a763d9c8b476afc67adfacb066051c7 not found: ID does not exist" Nov 03 23:06:47.022286 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:47.022232 4625 scope.go:122] "RemoveContainer" containerID="2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80" Nov 03 23:06:47.022492 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:47.022473 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80\": container with ID starting with 2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80 not found: ID does not exist" containerID="2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80" Nov 03 23:06:47.022535 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:47.022498 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80"} err="failed to get container status \"2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80\": rpc error: code = NotFound desc = could not find container \"2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80\": container with ID starting with 2f34108cc1456d65cb879846b302f78c62ac194596f7a9a0a17157841d420b80 not found: ID does not exist" Nov 03 23:06:47.043213 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:47.043176 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw"] Nov 03 23:06:47.046971 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:47.046943 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-npjdw"] Nov 03 23:06:48.727184 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:48.727154 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" path="/var/lib/kubelet/pods/e6725a80-3ec7-436a-a031-240f2c83e623/volumes" Nov 03 23:06:49.420466 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.420441 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 23:06:49.512031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.511945 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4fd4316-b749-438d-819d-1e05ef587b13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fd4316-b749-438d-819d-1e05ef587b13-istio-podinfo\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512071 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-kube-api-access-dqx7v\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-kube-api-access-dqx7v\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512107 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-credential-socket\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512142 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fd4316-b749-438d-819d-1e05ef587b13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fd4316-b749-438d-819d-1e05ef587b13-istiod-ca-cert\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512191 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-data\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512502 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512248 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-istio-token\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512502 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512281 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-envoy\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512502 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512337 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-certs\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512502 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512412 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-socket\") pod \"c4fd4316-b749-438d-819d-1e05ef587b13\" (UID: \"c4fd4316-b749-438d-819d-1e05ef587b13\") " Nov 03 23:06:49.512502 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512438 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-credential-socket" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:49.512798 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512609 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fd4316-b749-438d-819d-1e05ef587b13-istiod-ca-cert" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:49.512874 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512852 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-socket" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:49.512990 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512967 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.513051 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.512995 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.513051 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.513010 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fd4316-b749-438d-819d-1e05ef587b13-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.513279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.513257 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-certs" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:49.513411 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.513313 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-data" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:49.514581 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.514554 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-kube-api-access-dqx7v" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "kube-api-access-dqx7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:49.514972 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.514942 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-envoy" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:49.515126 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.515101 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-istio-token" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:49.515360 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.515331 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4fd4316-b749-438d-819d-1e05ef587b13-istio-podinfo" pod "c4fd4316-b749-438d-819d-1e05ef587b13" (UID: "c4fd4316-b749-438d-819d-1e05ef587b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:49.614007 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.613969 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqx7v\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-kube-api-access-dqx7v\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.614007 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.614001 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.614007 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.614012 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd4316-b749-438d-819d-1e05ef587b13-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.614244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.614020 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.614244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.614028 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd4316-b749-438d-819d-1e05ef587b13-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:49.614244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:49.614036 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fd4316-b749-438d-819d-1e05ef587b13-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:50.016197 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.016159 4625 generic.go:356] "Generic (PLEG): container finished" podID="c4fd4316-b749-438d-819d-1e05ef587b13" containerID="dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba" exitCode=0 Nov 03 23:06:50.016779 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.016228 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" Nov 03 23:06:50.016779 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.016232 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" event={"ID":"c4fd4316-b749-438d-819d-1e05ef587b13","Type":"ContainerDied","Data":"dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba"} Nov 03 23:06:50.016779 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.016293 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf" event={"ID":"c4fd4316-b749-438d-819d-1e05ef587b13","Type":"ContainerDied","Data":"8c9183bbbd57b61ea33ec7958af3fabc735be52301033152e0a63c4ddac01e74"} Nov 03 23:06:50.016779 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.016320 4625 scope.go:122] "RemoveContainer" containerID="dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba" Nov 03 23:06:50.027365 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.027346 4625 scope.go:122] "RemoveContainer" containerID="dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba" Nov 03 23:06:50.027671 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:06:50.027652 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba\": container with ID starting with dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba not found: ID does not exist" containerID="dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba" Nov 03 23:06:50.027738 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.027681 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba"} err="failed to get container status \"dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba\": rpc error: code = NotFound desc = could not find container \"dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba\": container with ID starting with dcf04b2afc9685fc14c57385b18b64bb1099b473229c0d332048f69d523788ba not found: ID does not exist" Nov 03 23:06:50.071865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.071831 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf"] Nov 03 23:06:50.083551 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.083495 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-2nggf"] Nov 03 23:06:50.728223 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:50.728185 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" path="/var/lib/kubelet/pods/c4fd4316-b749-438d-819d-1e05ef587b13/volumes" Nov 03 23:06:53.810764 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:53.810729 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b"] Nov 03 23:06:53.811214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:53.810985 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" containerID="cri-o://e32c5bbe286e89493e9514096522f046ce91736908dcfa8ef80c36ba391540a4" gracePeriod=30 Nov 03 23:06:59.089653 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.089615 4625 generic.go:356] "Generic (PLEG): container finished" podID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerID="e32c5bbe286e89493e9514096522f046ce91736908dcfa8ef80c36ba391540a4" exitCode=0 Nov 03 23:06:59.090164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.089749 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" event={"ID":"deb3babc-c24a-417f-9dd9-cc332cf6d463","Type":"ContainerDied","Data":"e32c5bbe286e89493e9514096522f046ce91736908dcfa8ef80c36ba391540a4"} Nov 03 23:06:59.090164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.089809 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" event={"ID":"deb3babc-c24a-417f-9dd9-cc332cf6d463","Type":"ContainerDied","Data":"ada7ff7add01a1da62a08a7a458cf5fc740d3677ed7c8663588af3992f44f953"} Nov 03 23:06:59.090164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.089836 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ada7ff7add01a1da62a08a7a458cf5fc740d3677ed7c8663588af3992f44f953" Nov 03 23:06:59.094879 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.094857 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 23:06:59.212156 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212129 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-data\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212172 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-token\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-token\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212217 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-credential-socket\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212236 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb3babc-c24a-417f-9dd9-cc332cf6d463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3babc-c24a-417f-9dd9-cc332cf6d463-istiod-ca-cert\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212256 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-certs\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212597 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212354 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-envoy\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212597 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212435 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-kube-api-access-z4vsd\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-kube-api-access-z4vsd\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212597 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212434 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-data" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:59.212597 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212465 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-podinfo\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.212597 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212475 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-credential-socket" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:59.212597 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212495 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-socket\") pod \"deb3babc-c24a-417f-9dd9-cc332cf6d463\" (UID: \"deb3babc-c24a-417f-9dd9-cc332cf6d463\") " Nov 03 23:06:59.213002 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212640 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-certs" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:59.213002 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212759 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb3babc-c24a-417f-9dd9-cc332cf6d463-istiod-ca-cert" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:59.213002 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.212895 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-socket" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:59.213377 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.213355 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.213485 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.213386 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.213485 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.213403 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.213485 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.213416 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3babc-c24a-417f-9dd9-cc332cf6d463-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.213485 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.213429 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.214673 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.214641 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-envoy" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:59.214827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.214681 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-token" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:59.214963 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.214941 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-podinfo" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:59.215033 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.214959 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-kube-api-access-z4vsd" pod "deb3babc-c24a-417f-9dd9-cc332cf6d463" (UID: "deb3babc-c24a-417f-9dd9-cc332cf6d463"). InnerVolumeSpecName "kube-api-access-z4vsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:59.314273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.314228 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.314273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.314271 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.314587 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.314289 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4vsd\" (UniqueName: \"kubernetes.io/projected/deb3babc-c24a-417f-9dd9-cc332cf6d463-kube-api-access-z4vsd\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.314587 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.314304 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3babc-c24a-417f-9dd9-cc332cf6d463-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:06:59.347602 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:06:59.347552 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2"] Nov 03 23:07:00.095417 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:00.095390 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b" Nov 03 23:07:00.183734 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:00.183685 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b"] Nov 03 23:07:00.207219 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:00.207175 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-xdg5b"] Nov 03 23:07:00.726974 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:00.726946 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" path="/var/lib/kubelet/pods/deb3babc-c24a-417f-9dd9-cc332cf6d463/volumes" Nov 03 23:07:02.185258 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:07:02.185216 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:07:03.873667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:03.873636 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x"] Nov 03 23:07:03.874084 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:03.873870 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" podUID="9949b744-8cff-4426-9f0d-0d3b36027091" containerName="maas-api" containerID="cri-o://d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37" gracePeriod=30 Nov 03 23:07:04.119747 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.119691 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 23:07:04.120962 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.120938 4625 generic.go:356] "Generic (PLEG): container finished" podID="9949b744-8cff-4426-9f0d-0d3b36027091" containerID="d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37" exitCode=0 Nov 03 23:07:04.121091 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.121012 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" event={"ID":"9949b744-8cff-4426-9f0d-0d3b36027091","Type":"ContainerDied","Data":"d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37"} Nov 03 23:07:04.121091 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.121079 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" event={"ID":"9949b744-8cff-4426-9f0d-0d3b36027091","Type":"ContainerDied","Data":"5b4678d5d89dfe3f7b9b91d7e9ae7b4cf06b0016c89ed33897e7ef4dc667881d"} Nov 03 23:07:04.121168 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.121107 4625 scope.go:122] "RemoveContainer" containerID="d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37" Nov 03 23:07:04.147298 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.147251 4625 scope.go:122] "RemoveContainer" containerID="d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37" Nov 03 23:07:04.147711 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:07:04.147682 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37\": container with ID starting with d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37 not found: ID does not exist" containerID="d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37" Nov 03 23:07:04.147832 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.147751 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37"} err="failed to get container status \"d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37\": rpc error: code = NotFound desc = could not find container \"d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37\": container with ID starting with d25fd88bd4a71cd847c1f4aa3cb351c91803cd795cc12908a3081752e2711e37 not found: ID does not exist" Nov 03 23:07:04.277088 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.277053 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-metrics-certs\") pod \"9949b744-8cff-4426-9f0d-0d3b36027091\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " Nov 03 23:07:04.277263 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.277123 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9949b744-8cff-4426-9f0d-0d3b36027091-kube-api-access-m9qnl\" (UniqueName: \"kubernetes.io/projected/9949b744-8cff-4426-9f0d-0d3b36027091-kube-api-access-m9qnl\") pod \"9949b744-8cff-4426-9f0d-0d3b36027091\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " Nov 03 23:07:04.277263 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.277154 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-maas-api-tls\") pod \"9949b744-8cff-4426-9f0d-0d3b36027091\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " Nov 03 23:07:04.277263 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.277183 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9949b744-8cff-4426-9f0d-0d3b36027091-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949b744-8cff-4426-9f0d-0d3b36027091-openshift-service-ca\") pod \"9949b744-8cff-4426-9f0d-0d3b36027091\" (UID: \"9949b744-8cff-4426-9f0d-0d3b36027091\") " Nov 03 23:07:04.277662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.277599 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9949b744-8cff-4426-9f0d-0d3b36027091-openshift-service-ca" pod "9949b744-8cff-4426-9f0d-0d3b36027091" (UID: "9949b744-8cff-4426-9f0d-0d3b36027091"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:04.279761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.279731 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9949b744-8cff-4426-9f0d-0d3b36027091-kube-api-access-m9qnl" pod "9949b744-8cff-4426-9f0d-0d3b36027091" (UID: "9949b744-8cff-4426-9f0d-0d3b36027091"). InnerVolumeSpecName "kube-api-access-m9qnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:04.279886 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.279756 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-metrics-certs" pod "9949b744-8cff-4426-9f0d-0d3b36027091" (UID: "9949b744-8cff-4426-9f0d-0d3b36027091"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:04.280160 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.280140 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-maas-api-tls" pod "9949b744-8cff-4426-9f0d-0d3b36027091" (UID: "9949b744-8cff-4426-9f0d-0d3b36027091"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:04.379437 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.379388 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:04.382071 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.382038 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9qnl\" (UniqueName: \"kubernetes.io/projected/9949b744-8cff-4426-9f0d-0d3b36027091-kube-api-access-m9qnl\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:04.382333 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.382074 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949b744-8cff-4426-9f0d-0d3b36027091-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:04.382333 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:04.382095 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949b744-8cff-4426-9f0d-0d3b36027091-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:05.040532 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:05.040443 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs"] Nov 03 23:07:05.131497 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:05.131455 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x" Nov 03 23:07:05.194990 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:05.194944 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x"] Nov 03 23:07:05.204934 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:05.204889 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d8a3d1-6cccd6c557-q4c9x"] Nov 03 23:07:06.727334 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:06.727293 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9949b744-8cff-4426-9f0d-0d3b36027091" path="/var/lib/kubelet/pods/9949b744-8cff-4426-9f0d-0d3b36027091/volumes" Nov 03 23:07:07.471224 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.471191 4625 scope.go:122] "RemoveContainer" containerID="e32c5bbe286e89493e9514096522f046ce91736908dcfa8ef80c36ba391540a4" Nov 03 23:07:07.913158 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913128 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9"] Nov 03 23:07:07.913578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913565 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" Nov 03 23:07:07.913578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913577 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913592 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f70a8fa5-192c-486d-825f-d45c9342c6ca" containerName="maas-api" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913598 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70a8fa5-192c-486d-825f-d45c9342c6ca" containerName="maas-api" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913606 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2d07e94e-9f07-4af8-b8fb-1d316bd5705f" containerName="maas-api" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913612 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d07e94e-9f07-4af8-b8fb-1d316bd5705f" containerName="maas-api" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913621 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7fc247d5-97b3-4b2d-97e8-e08346346447" containerName="payload-pre-processing" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913626 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc247d5-97b3-4b2d-97e8-e08346346447" containerName="payload-pre-processing" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913634 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913641 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913652 4625 cpu_manager.go:414] "containerMap: removing container" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913656 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913664 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913669 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" Nov 03 23:07:07.913667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913675 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913680 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913687 4625 cpu_manager.go:414] "containerMap: removing container" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913692 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913698 4625 cpu_manager.go:414] "containerMap: removing container" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913702 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913709 4625 cpu_manager.go:414] "containerMap: removing container" podUID="4ed4df38-d21b-4b49-a0c8-076f51459320" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913728 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed4df38-d21b-4b49-a0c8-076f51459320" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913740 4625 cpu_manager.go:414] "containerMap: removing container" podUID="5f1eacc9-48d4-437b-9421-b451ed4f7ebd" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913748 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1eacc9-48d4-437b-9421-b451ed4f7ebd" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913756 4625 cpu_manager.go:414] "containerMap: removing container" podUID="ac4ab126-b50f-4673-8ff5-12444e47b637" containerName="payload-processing" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913761 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4ab126-b50f-4673-8ff5-12444e47b637" containerName="payload-processing" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913771 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913777 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913789 4625 cpu_manager.go:414] "containerMap: removing container" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913796 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913806 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913811 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913818 4625 cpu_manager.go:414] "containerMap: removing container" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913823 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913829 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d2360c3d-31be-4438-aa78-24489e87bb66" containerName="payload-pre-processing" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913833 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2360c3d-31be-4438-aa78-24489e87bb66" containerName="payload-pre-processing" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913840 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913844 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913853 4625 cpu_manager.go:414] "containerMap: removing container" podUID="676907ab-2cf3-4909-810b-848642fe6bb7" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913860 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676907ab-2cf3-4909-810b-848642fe6bb7" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913869 4625 cpu_manager.go:414] "containerMap: removing container" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913874 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913880 4625 cpu_manager.go:414] "containerMap: removing container" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913884 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913890 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3123211a-c11f-4479-be91-2bf9edcb267e" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913894 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3123211a-c11f-4479-be91-2bf9edcb267e" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913900 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913904 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913912 4625 cpu_manager.go:414] "containerMap: removing container" podUID="701abc1a-231d-42a0-a997-ba3e21759e37" containerName="payload-processing" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913917 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701abc1a-231d-42a0-a997-ba3e21759e37" containerName="payload-processing" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913925 4625 cpu_manager.go:414] "containerMap: removing container" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913929 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913935 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913939 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913944 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9949b744-8cff-4426-9f0d-0d3b36027091" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913949 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9949b744-8cff-4426-9f0d-0d3b36027091" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913956 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b1996cfe-4286-48f4-9db9-93454203b234" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913960 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1996cfe-4286-48f4-9db9-93454203b234" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913965 4625 cpu_manager.go:414] "containerMap: removing container" podUID="18fe9477-aa04-483c-9d15-39df7f73c174" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913971 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18fe9477-aa04-483c-9d15-39df7f73c174" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913978 4625 cpu_manager.go:414] "containerMap: removing container" podUID="1c53996e-be1a-4df1-a7c7-4bd4daecda71" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913982 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c53996e-be1a-4df1-a7c7-4bd4daecda71" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913989 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.913994 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914001 4625 cpu_manager.go:414] "containerMap: removing container" podUID="431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914005 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914012 4625 cpu_manager.go:414] "containerMap: removing container" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914016 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914023 4625 cpu_manager.go:414] "containerMap: removing container" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914028 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914035 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914040 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914047 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e0217efd-c387-423c-8e37-a4dc7c89d36e" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914051 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0217efd-c387-423c-8e37-a4dc7c89d36e" containerName="maas-api" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914059 4625 cpu_manager.go:414] "containerMap: removing container" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914064 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914072 4625 cpu_manager.go:414] "containerMap: removing container" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" Nov 03 23:07:07.914115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914076 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914161 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1996cfe-4286-48f4-9db9-93454203b234" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914170 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="18fe9477-aa04-483c-9d15-39df7f73c174" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914176 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c53996e-be1a-4df1-a7c7-4bd4daecda71" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914183 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30970e3-1702-45fa-bc60-bcd754afa716" containerName="main" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914187 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="431b53a2-7fb3-42eb-9e1f-6ac36a1dc51a" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914194 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de0a5b4-30e8-440b-8a10-832f72c5e885" containerName="main" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914200 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb3babc-c24a-417f-9dd9-cc332cf6d463" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914205 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b91c160-5f2d-43b6-bf6f-b0c593b1a3c9" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914212 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0217efd-c387-423c-8e37-a4dc7c89d36e" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914218 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914223 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914228 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="427a0694-0bae-463c-a121-3f84d3e01ada" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914238 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fd4316-b749-438d-819d-1e05ef587b13" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914247 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70a8fa5-192c-486d-825f-d45c9342c6ca" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914252 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d07e94e-9f07-4af8-b8fb-1d316bd5705f" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914257 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc247d5-97b3-4b2d-97e8-e08346346447" containerName="payload-pre-processing" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914263 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914270 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914275 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9903686-a262-41cd-b4fd-2bf5936cddca" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914281 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914286 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914290 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed4df38-d21b-4b49-a0c8-076f51459320" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914295 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1eacc9-48d4-437b-9421-b451ed4f7ebd" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914299 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4ab126-b50f-4673-8ff5-12444e47b637" containerName="payload-processing" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914304 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914310 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d2fe03-81da-460a-ba49-df3afe3eefee" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914315 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914319 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914325 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2360c3d-31be-4438-aa78-24489e87bb66" containerName="payload-pre-processing" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914330 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0883f5-f7f0-49d2-b29c-56cc1c03e0b3" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914335 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="676907ab-2cf3-4909-810b-848642fe6bb7" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914339 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ceb83ca-1c71-40d1-a181-88d931f3538d" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914344 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="25766c62-3e88-493d-bfeb-a3e33b86e67c" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914350 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3123211a-c11f-4479-be91-2bf9edcb267e" containerName="maas-api" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914355 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f93335b-e263-455b-a2eb-d34acc1f1398" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914360 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="701abc1a-231d-42a0-a997-ba3e21759e37" containerName="payload-processing" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914364 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dabd04-7e6e-46cd-9331-ced0975c71ca" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914370 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d076d7-0968-4cb1-822e-b1bacd73c7db" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914374 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" Nov 03 23:07:07.916384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.914380 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="9949b744-8cff-4426-9f0d-0d3b36027091" containerName="maas-api" Nov 03 23:07:07.919950 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.919931 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:07.939726 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:07.939547 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9"] Nov 03 23:07:08.013148 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.013107 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:08.013333 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.013170 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:07:08.016654 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.016632 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.016776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.016672 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.016845 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.016826 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrf5\" (UniqueName: \"kubernetes.io/projected/6ee74ee7-e9fa-4677-9f25-174440a86b5d-kube-api-access-wwrf5\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.118241 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.118205 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrf5\" (UniqueName: \"kubernetes.io/projected/6ee74ee7-e9fa-4677-9f25-174440a86b5d-kube-api-access-wwrf5\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.118426 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.118302 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.118426 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.118333 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.119001 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.118981 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.120913 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.120887 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.143775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.143706 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrf5\" (UniqueName: \"kubernetes.io/projected/6ee74ee7-e9fa-4677-9f25-174440a86b5d-kube-api-access-wwrf5\") pod \"maas-api-revoke-sub-789f5be1-7nrb9\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.233805 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.233705 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:08.551235 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:08.551210 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9"] Nov 03 23:07:08.553912 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:07:08.553873 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee74ee7_e9fa_4677_9f25_174440a86b5d.slice/crio-5c118d49344e7330c9a4a0d42142ad547c8ffdda1e8df1befa4754e5422b2d5d WatchSource:0}: Error finding container 5c118d49344e7330c9a4a0d42142ad547c8ffdda1e8df1befa4754e5422b2d5d: Status 404 returned error can't find the container with id 5c118d49344e7330c9a4a0d42142ad547c8ffdda1e8df1befa4754e5422b2d5d Nov 03 23:07:09.158675 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:09.158633 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" event={"ID":"6ee74ee7-e9fa-4677-9f25-174440a86b5d","Type":"ContainerStarted","Data":"8f2d736972405141a210d7e442b2ef83084929d574d8ca17ec4eebb4247d63e6"} Nov 03 23:07:09.159110 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:09.158699 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" event={"ID":"6ee74ee7-e9fa-4677-9f25-174440a86b5d","Type":"ContainerStarted","Data":"5c118d49344e7330c9a4a0d42142ad547c8ffdda1e8df1befa4754e5422b2d5d"} Nov 03 23:07:09.199931 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:09.199860 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" podStartSLOduration=2.199841993 podStartE2EDuration="2.199841993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:09.198070284 +0000 UTC m=+3250.627406277" watchObservedRunningTime="2026-11-03 23:07:09.199841993 +0000 UTC m=+3250.629177991" Nov 03 23:07:10.041209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.041167 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" podUID="36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" containerName="payload-processing" containerID="cri-o://ddfba8271845e9ab73a90972e1c866c650a3156d0323f742a5d68abad11e2813" gracePeriod=25 Nov 03 23:07:10.166151 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.166114 4625 generic.go:356] "Generic (PLEG): container finished" podID="6ee74ee7-e9fa-4677-9f25-174440a86b5d" containerID="8f2d736972405141a210d7e442b2ef83084929d574d8ca17ec4eebb4247d63e6" exitCode=0 Nov 03 23:07:10.166586 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.166194 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" event={"ID":"6ee74ee7-e9fa-4677-9f25-174440a86b5d","Type":"ContainerDied","Data":"8f2d736972405141a210d7e442b2ef83084929d574d8ca17ec4eebb4247d63e6"} Nov 03 23:07:10.167938 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.167913 4625 generic.go:356] "Generic (PLEG): container finished" podID="36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" containerID="ddfba8271845e9ab73a90972e1c866c650a3156d0323f742a5d68abad11e2813" exitCode=0 Nov 03 23:07:10.168085 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.167961 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" event={"ID":"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54","Type":"ContainerDied","Data":"ddfba8271845e9ab73a90972e1c866c650a3156d0323f742a5d68abad11e2813"} Nov 03 23:07:10.304580 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.304516 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 23:07:10.442847 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.442802 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-plugins-config-volume\") pod \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " Nov 03 23:07:10.443022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.442918 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-kube-api-access-ltm82\" (UniqueName: \"kubernetes.io/projected/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-kube-api-access-ltm82\") pod \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\" (UID: \"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54\") " Nov 03 23:07:10.443308 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.443276 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-plugins-config-volume" pod "36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" (UID: "36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:10.445142 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.445117 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-kube-api-access-ltm82" pod "36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" (UID: "36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54"). InnerVolumeSpecName "kube-api-access-ltm82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:10.544704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.544662 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltm82\" (UniqueName: \"kubernetes.io/projected/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-kube-api-access-ltm82\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:10.544704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:10.544701 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:11.177066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.177029 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" Nov 03 23:07:11.177066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.177060 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs" event={"ID":"36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54","Type":"ContainerDied","Data":"9b2ae4d17a10dbc4a3c52e2950e6b9e74e006d08eb2c8d96936f1b88b12fd9ea"} Nov 03 23:07:11.177741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.177126 4625 scope.go:122] "RemoveContainer" containerID="ddfba8271845e9ab73a90972e1c866c650a3156d0323f742a5d68abad11e2813" Nov 03 23:07:11.231581 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.231547 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs"] Nov 03 23:07:11.248320 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.248284 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-88a4d9-76c4574f8b-g8sjs"] Nov 03 23:07:11.309541 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.309513 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:11.455608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.455508 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee74ee7-e9fa-4677-9f25-174440a86b5d-kube-api-access-wwrf5\" (UniqueName: \"kubernetes.io/projected/6ee74ee7-e9fa-4677-9f25-174440a86b5d-kube-api-access-wwrf5\") pod \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " Nov 03 23:07:11.455608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.455600 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-xks-ca\") pod \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " Nov 03 23:07:11.455876 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.455707 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-openshift-ca\") pod \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\" (UID: \"6ee74ee7-e9fa-4677-9f25-174440a86b5d\") " Nov 03 23:07:11.456303 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.456271 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-openshift-ca" pod "6ee74ee7-e9fa-4677-9f25-174440a86b5d" (UID: "6ee74ee7-e9fa-4677-9f25-174440a86b5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:11.457762 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.457704 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-xks-ca" pod "6ee74ee7-e9fa-4677-9f25-174440a86b5d" (UID: "6ee74ee7-e9fa-4677-9f25-174440a86b5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:11.457896 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.457877 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee74ee7-e9fa-4677-9f25-174440a86b5d-kube-api-access-wwrf5" pod "6ee74ee7-e9fa-4677-9f25-174440a86b5d" (UID: "6ee74ee7-e9fa-4677-9f25-174440a86b5d"). InnerVolumeSpecName "kube-api-access-wwrf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:11.556994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.556955 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:11.556994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.556995 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwrf5\" (UniqueName: \"kubernetes.io/projected/6ee74ee7-e9fa-4677-9f25-174440a86b5d-kube-api-access-wwrf5\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:11.557236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:11.557015 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee74ee7-e9fa-4677-9f25-174440a86b5d-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:12.185904 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:12.185875 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" Nov 03 23:07:12.186344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:12.185870 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9" event={"ID":"6ee74ee7-e9fa-4677-9f25-174440a86b5d","Type":"ContainerDied","Data":"5c118d49344e7330c9a4a0d42142ad547c8ffdda1e8df1befa4754e5422b2d5d"} Nov 03 23:07:12.186344 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:12.186006 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c118d49344e7330c9a4a0d42142ad547c8ffdda1e8df1befa4754e5422b2d5d" Nov 03 23:07:12.731978 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:12.731941 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" path="/var/lib/kubelet/pods/36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54/volumes" Nov 03 23:07:12.732396 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:12.732377 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6"] Nov 03 23:07:12.732877 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:12.732846 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" containerID="cri-o://efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9" gracePeriod=30 Nov 03 23:07:13.168842 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:13.168802 4625 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:07:13.169065 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:13.168868 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:07:14.369705 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.369666 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" containerID="cri-o://a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9" gracePeriod=45 Nov 03 23:07:14.626079 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.626009 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 23:07:14.697403 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697358 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-home\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-home\") pod \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " Nov 03 23:07:14.697582 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697446 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tmp-dir\") pod \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " Nov 03 23:07:14.697582 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697516 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tls-certs\") pod \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " Nov 03 23:07:14.697702 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697592 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-kube-api-access-9s7h9\" (UniqueName: \"kubernetes.io/projected/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-kube-api-access-9s7h9\") pod \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " Nov 03 23:07:14.697702 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697624 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-home" pod "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" (UID: "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.697702 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697696 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-model-cache\") pod \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " Nov 03 23:07:14.697924 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697754 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-dshm\") pod \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\" (UID: \"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858\") " Nov 03 23:07:14.697924 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697765 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tmp-dir" pod "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" (UID: "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.698063 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.697924 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-model-cache" pod "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" (UID: "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.698124 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.698097 4625 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.698124 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.698112 4625 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.698124 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.698120 4625 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.700601 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.700510 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-dshm" pod "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" (UID: "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.700601 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.700563 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tls-certs" pod "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" (UID: "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:14.700601 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.699994 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-kube-api-access-9s7h9" pod "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" (UID: "0d2a6a8c-1a88-4ecf-9a48-f06b18e51858"). InnerVolumeSpecName "kube-api-access-9s7h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:14.799552 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.799515 4625 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.799552 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.799544 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7h9\" (UniqueName: \"kubernetes.io/projected/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-kube-api-access-9s7h9\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.799552 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:14.799557 4625 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:15.203836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.203804 4625 generic.go:356] "Generic (PLEG): container finished" podID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerID="a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9" exitCode=0 Nov 03 23:07:15.204022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.203854 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" Nov 03 23:07:15.204022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.203892 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" event={"ID":"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858","Type":"ContainerDied","Data":"a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9"} Nov 03 23:07:15.204022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.203937 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2" event={"ID":"0d2a6a8c-1a88-4ecf-9a48-f06b18e51858","Type":"ContainerDied","Data":"cd5c0f110dc39e7a48c190fdb223392b0d63db04b1e4bd375ff2dcb479e28304"} Nov 03 23:07:15.204022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.203956 4625 scope.go:122] "RemoveContainer" containerID="a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9" Nov 03 23:07:15.212827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.212809 4625 scope.go:122] "RemoveContainer" containerID="a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9" Nov 03 23:07:15.213096 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:07:15.213075 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9\": container with ID starting with a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9 not found: ID does not exist" containerID="a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9" Nov 03 23:07:15.213146 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.213105 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9"} err="failed to get container status \"a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9\": rpc error: code = NotFound desc = could not find container \"a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9\": container with ID starting with a5377dc9042184fc2e9feb2e3ccf91abd34f69477ccd9e693987c5918a387cf9 not found: ID does not exist" Nov 03 23:07:15.279191 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.279154 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2"] Nov 03 23:07:15.290319 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:15.290287 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-w0-88a4d9-kserve-84b7f99547-bktt2"] Nov 03 23:07:16.729534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:16.729493 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" path="/var/lib/kubelet/pods/0d2a6a8c-1a88-4ecf-9a48-f06b18e51858/volumes" Nov 03 23:07:17.990135 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:17.990104 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 23:07:18.035747 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.035271 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-podinfo\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.035747 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.035354 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ab7891-a909-4113-8881-0dd30efd2d0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ab7891-a909-4113-8881-0dd30efd2d0c-istiod-ca-cert\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.035969 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.035405 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-credential-socket\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.035969 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.035903 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-socket\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.036167 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036137 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-credential-socket" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.036315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036174 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-socket" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.036315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036201 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ab7891-a909-4113-8881-0dd30efd2d0c-istiod-ca-cert" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:18.036315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036238 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-envoy\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.036315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036297 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-kube-api-access-7dk59\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-kube-api-access-7dk59\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.036642 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036373 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-token\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-token\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.036642 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036410 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-data\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.036642 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036447 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-certs\") pod \"b3ab7891-a909-4113-8881-0dd30efd2d0c\" (UID: \"b3ab7891-a909-4113-8881-0dd30efd2d0c\") " Nov 03 23:07:18.037202 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036892 4625 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ab7891-a909-4113-8881-0dd30efd2d0c-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.037202 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036915 4625 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.037202 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.036927 4625 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.037481 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.037328 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-data" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.037481 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.037463 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-certs" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.038209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.038181 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-podinfo" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:18.038685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.038660 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-envoy" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.039575 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.039553 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-kube-api-access-7dk59" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "kube-api-access-7dk59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:18.040312 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.039896 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-token" pod "b3ab7891-a909-4113-8881-0dd30efd2d0c" (UID: "b3ab7891-a909-4113-8881-0dd30efd2d0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:18.137786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.137754 4625 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.137786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.137779 4625 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.137786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.137788 4625 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.137786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.137796 4625 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.138069 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.137804 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dk59\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-kube-api-access-7dk59\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.138069 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.137816 4625 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ab7891-a909-4113-8881-0dd30efd2d0c-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.230816 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.230773 4625 generic.go:356] "Generic (PLEG): container finished" podID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerID="efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9" exitCode=0 Nov 03 23:07:18.231016 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.230820 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" Nov 03 23:07:18.231016 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.230844 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" event={"ID":"b3ab7891-a909-4113-8881-0dd30efd2d0c","Type":"ContainerDied","Data":"efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9"} Nov 03 23:07:18.231016 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.230895 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6" event={"ID":"b3ab7891-a909-4113-8881-0dd30efd2d0c","Type":"ContainerDied","Data":"739165cf54c222547de630f514f0f8204fa7cef9fe765f38935dcfd0c361056b"} Nov 03 23:07:18.231016 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.230916 4625 scope.go:122] "RemoveContainer" containerID="efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9" Nov 03 23:07:18.241304 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.241287 4625 scope.go:122] "RemoveContainer" containerID="e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e" Nov 03 23:07:18.250123 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.250099 4625 scope.go:122] "RemoveContainer" containerID="efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9" Nov 03 23:07:18.250386 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:07:18.250363 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9\": container with ID starting with efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9 not found: ID does not exist" containerID="efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9" Nov 03 23:07:18.250497 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.250397 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9"} err="failed to get container status \"efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9\": rpc error: code = NotFound desc = could not find container \"efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9\": container with ID starting with efc7b917d1e5f22d8050c664ecc704a8e9ef4ca1696c4b6dff98f6a0d23306a9 not found: ID does not exist" Nov 03 23:07:18.250497 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.250420 4625 scope.go:122] "RemoveContainer" containerID="e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e" Nov 03 23:07:18.250674 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:07:18.250655 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e\": container with ID starting with e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e not found: ID does not exist" containerID="e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e" Nov 03 23:07:18.250734 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.250682 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e"} err="failed to get container status \"e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e\": rpc error: code = NotFound desc = could not find container \"e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e\": container with ID starting with e7298ff1b7ebe13f762db201595b6744e3f9a6b49b8ff4458b43533524e6c37e not found: ID does not exist" Nov 03 23:07:18.281651 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.281615 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6"] Nov 03 23:07:18.293683 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.293586 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-8q4n6"] Nov 03 23:07:18.727263 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:18.727232 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" path="/var/lib/kubelet/pods/b3ab7891-a909-4113-8881-0dd30efd2d0c/volumes" Nov 03 23:07:24.196585 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.196550 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z"] Nov 03 23:07:24.197027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.196990 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" Nov 03 23:07:24.197027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197002 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" Nov 03 23:07:24.197027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197021 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" Nov 03 23:07:24.197027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197028 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197035 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6ee74ee7-e9fa-4677-9f25-174440a86b5d" containerName="revoke-keys" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197040 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee74ee7-e9fa-4677-9f25-174440a86b5d" containerName="revoke-keys" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197050 4625 cpu_manager.go:414] "containerMap: removing container" podUID="36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" containerName="payload-processing" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197054 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" containerName="payload-processing" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197061 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197065 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197092 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197100 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6725a80-3ec7-436a-a031-240f2c83e623" containerName="istio-proxy" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197109 4625 cpu_manager.go:414] "containerMap: removing container" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" Nov 03 23:07:24.197174 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197114 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904859f4-8b39-48ed-8289-283eeedec8bd" containerName="istio-proxy" Nov 03 23:07:24.197481 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197179 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" Nov 03 23:07:24.197481 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197187 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2a6a8c-1a88-4ecf-9a48-f06b18e51858" containerName="main" Nov 03 23:07:24.197481 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197193 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee74ee7-e9fa-4677-9f25-174440a86b5d" containerName="revoke-keys" Nov 03 23:07:24.197481 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197200 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ac8d7a-9eb7-4015-ac4f-fbea81bcbd54" containerName="payload-processing" Nov 03 23:07:24.197481 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.197207 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="45159818-f7b5-45a8-b6b8-47873dfd02da" containerName="istio-proxy" Nov 03 23:07:24.212936 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.212904 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.221029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.220803 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z"] Nov 03 23:07:24.260180 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.260145 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:07:24.260396 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.260226 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:24.295812 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.295773 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.296004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.295903 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.296053 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.296022 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/00e36eab-2abd-4a73-b86d-be125deaa935-kube-api-access-2xwmg\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.396903 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.396861 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.397123 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.396965 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/00e36eab-2abd-4a73-b86d-be125deaa935-kube-api-access-2xwmg\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.397123 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.397000 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.397652 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.397626 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.399416 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.399390 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.420172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.420141 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/00e36eab-2abd-4a73-b86d-be125deaa935-kube-api-access-2xwmg\") pod \"maas-api-revoke-sub-5e22966d-s9z5z\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.530025 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.529933 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:24.844611 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:24.844584 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z"] Nov 03 23:07:24.848314 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:07:24.848279 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e36eab_2abd_4a73_b86d_be125deaa935.slice/crio-e4353edee17bfcbb7efa49e6c98f567db9813ef9514787e789c684f1f4b801b5 WatchSource:0}: Error finding container e4353edee17bfcbb7efa49e6c98f567db9813ef9514787e789c684f1f4b801b5: Status 404 returned error can't find the container with id e4353edee17bfcbb7efa49e6c98f567db9813ef9514787e789c684f1f4b801b5 Nov 03 23:07:25.276256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:25.276219 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" event={"ID":"00e36eab-2abd-4a73-b86d-be125deaa935","Type":"ContainerStarted","Data":"9c96bf8b660e405875a251ab88b2d708d471130e70c296bf4688f6d00aac46bd"} Nov 03 23:07:25.276680 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:25.276279 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" event={"ID":"00e36eab-2abd-4a73-b86d-be125deaa935","Type":"ContainerStarted","Data":"e4353edee17bfcbb7efa49e6c98f567db9813ef9514787e789c684f1f4b801b5"} Nov 03 23:07:25.318839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:25.318785 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" podStartSLOduration=1.318766632 podStartE2EDuration="1.318766632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:25.316278806 +0000 UTC m=+3266.745614802" watchObservedRunningTime="2026-11-03 23:07:25.318766632 +0000 UTC m=+3266.748102624" Nov 03 23:07:26.284826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:26.284787 4625 generic.go:356] "Generic (PLEG): container finished" podID="00e36eab-2abd-4a73-b86d-be125deaa935" containerID="9c96bf8b660e405875a251ab88b2d708d471130e70c296bf4688f6d00aac46bd" exitCode=0 Nov 03 23:07:26.285261 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:26.284851 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" event={"ID":"00e36eab-2abd-4a73-b86d-be125deaa935","Type":"ContainerDied","Data":"9c96bf8b660e405875a251ab88b2d708d471130e70c296bf4688f6d00aac46bd"} Nov 03 23:07:27.437067 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.437036 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:27.526795 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.526763 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e36eab-2abd-4a73-b86d-be125deaa935-kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/00e36eab-2abd-4a73-b86d-be125deaa935-kube-api-access-2xwmg\") pod \"00e36eab-2abd-4a73-b86d-be125deaa935\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " Nov 03 23:07:27.527027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.526850 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-openshift-ca\") pod \"00e36eab-2abd-4a73-b86d-be125deaa935\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " Nov 03 23:07:27.527027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.526943 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-xks-ca\") pod \"00e36eab-2abd-4a73-b86d-be125deaa935\" (UID: \"00e36eab-2abd-4a73-b86d-be125deaa935\") " Nov 03 23:07:27.527224 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.527203 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-openshift-ca" pod "00e36eab-2abd-4a73-b86d-be125deaa935" (UID: "00e36eab-2abd-4a73-b86d-be125deaa935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:27.529010 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.528987 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-xks-ca" pod "00e36eab-2abd-4a73-b86d-be125deaa935" (UID: "00e36eab-2abd-4a73-b86d-be125deaa935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:27.529119 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.529100 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e36eab-2abd-4a73-b86d-be125deaa935-kube-api-access-2xwmg" pod "00e36eab-2abd-4a73-b86d-be125deaa935" (UID: "00e36eab-2abd-4a73-b86d-be125deaa935"). InnerVolumeSpecName "kube-api-access-2xwmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:27.627906 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.627870 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:27.627906 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.627900 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36eab-2abd-4a73-b86d-be125deaa935-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:27.627906 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:27.627912 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/00e36eab-2abd-4a73-b86d-be125deaa935-kube-api-access-2xwmg\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:28.297569 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:28.297537 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" Nov 03 23:07:28.297785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:28.297570 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z" event={"ID":"00e36eab-2abd-4a73-b86d-be125deaa935","Type":"ContainerDied","Data":"e4353edee17bfcbb7efa49e6c98f567db9813ef9514787e789c684f1f4b801b5"} Nov 03 23:07:28.297785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:28.297617 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4353edee17bfcbb7efa49e6c98f567db9813ef9514787e789c684f1f4b801b5" Nov 03 23:07:32.185778 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:07:32.185741 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:07:36.275859 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.275823 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq"] Nov 03 23:07:36.276351 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.276031 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" podUID="42fac85f-2514-47c1-931a-37ccc7b40a6e" containerName="maas-api" containerID="cri-o://60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e" gracePeriod=30 Nov 03 23:07:36.539024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.538680 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 23:07:36.618686 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.618656 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-metrics-certs\") pod \"42fac85f-2514-47c1-931a-37ccc7b40a6e\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " Nov 03 23:07:36.618883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.618773 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fac85f-2514-47c1-931a-37ccc7b40a6e-kube-api-access-7kxnt\" (UniqueName: \"kubernetes.io/projected/42fac85f-2514-47c1-931a-37ccc7b40a6e-kube-api-access-7kxnt\") pod \"42fac85f-2514-47c1-931a-37ccc7b40a6e\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " Nov 03 23:07:36.618883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.618806 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fac85f-2514-47c1-931a-37ccc7b40a6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fac85f-2514-47c1-931a-37ccc7b40a6e-openshift-service-ca\") pod \"42fac85f-2514-47c1-931a-37ccc7b40a6e\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " Nov 03 23:07:36.618883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.618865 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls\") pod \"42fac85f-2514-47c1-931a-37ccc7b40a6e\" (UID: \"42fac85f-2514-47c1-931a-37ccc7b40a6e\") " Nov 03 23:07:36.619249 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.619222 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fac85f-2514-47c1-931a-37ccc7b40a6e-openshift-service-ca" pod "42fac85f-2514-47c1-931a-37ccc7b40a6e" (UID: "42fac85f-2514-47c1-931a-37ccc7b40a6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:36.621018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.620990 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-metrics-certs" pod "42fac85f-2514-47c1-931a-37ccc7b40a6e" (UID: "42fac85f-2514-47c1-931a-37ccc7b40a6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:36.621130 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.621117 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls" pod "42fac85f-2514-47c1-931a-37ccc7b40a6e" (UID: "42fac85f-2514-47c1-931a-37ccc7b40a6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:36.621212 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.621198 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fac85f-2514-47c1-931a-37ccc7b40a6e-kube-api-access-7kxnt" pod "42fac85f-2514-47c1-931a-37ccc7b40a6e" (UID: "42fac85f-2514-47c1-931a-37ccc7b40a6e"). InnerVolumeSpecName "kube-api-access-7kxnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:36.719856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.719816 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:36.719856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.719854 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fac85f-2514-47c1-931a-37ccc7b40a6e-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:36.720140 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.719870 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kxnt\" (UniqueName: \"kubernetes.io/projected/42fac85f-2514-47c1-931a-37ccc7b40a6e-kube-api-access-7kxnt\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:36.720140 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:36.719885 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fac85f-2514-47c1-931a-37ccc7b40a6e-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:07:37.352743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.352674 4625 generic.go:356] "Generic (PLEG): container finished" podID="42fac85f-2514-47c1-931a-37ccc7b40a6e" containerID="60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e" exitCode=0 Nov 03 23:07:37.353256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.352766 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" Nov 03 23:07:37.353256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.352763 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" event={"ID":"42fac85f-2514-47c1-931a-37ccc7b40a6e","Type":"ContainerDied","Data":"60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e"} Nov 03 23:07:37.353256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.352899 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq" event={"ID":"42fac85f-2514-47c1-931a-37ccc7b40a6e","Type":"ContainerDied","Data":"d84368cf45b04daa9b583976fac67f5086b40a65c28b7d509ee94150509aca82"} Nov 03 23:07:37.353256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.352925 4625 scope.go:122] "RemoveContainer" containerID="60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e" Nov 03 23:07:37.363785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.363755 4625 scope.go:122] "RemoveContainer" containerID="60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e" Nov 03 23:07:37.364068 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:07:37.364050 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e\": container with ID starting with 60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e not found: ID does not exist" containerID="60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e" Nov 03 23:07:37.364136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.364081 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e"} err="failed to get container status \"60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e\": rpc error: code = NotFound desc = could not find container \"60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e\": container with ID starting with 60fae2b364c07bcb6b4c0195840b68052028e0507494750e00318c19074d0c7e not found: ID does not exist" Nov 03 23:07:37.403676 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.403639 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq"] Nov 03 23:07:37.416256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:37.416228 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5187e1-6774fb747-np9jq"] Nov 03 23:07:38.726156 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:38.726126 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fac85f-2514-47c1-931a-37ccc7b40a6e" path="/var/lib/kubelet/pods/42fac85f-2514-47c1-931a-37ccc7b40a6e/volumes" Nov 03 23:07:51.320529 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.320442 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9589f759-tfgvc"] Nov 03 23:07:51.321115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.321096 4625 cpu_manager.go:414] "containerMap: removing container" podUID="42fac85f-2514-47c1-931a-37ccc7b40a6e" containerName="maas-api" Nov 03 23:07:51.321169 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.321119 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fac85f-2514-47c1-931a-37ccc7b40a6e" containerName="maas-api" Nov 03 23:07:51.321207 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.321175 4625 cpu_manager.go:414] "containerMap: removing container" podUID="00e36eab-2abd-4a73-b86d-be125deaa935" containerName="revoke-keys" Nov 03 23:07:51.321207 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.321185 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e36eab-2abd-4a73-b86d-be125deaa935" containerName="revoke-keys" Nov 03 23:07:51.321294 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.321282 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ab7891-a909-4113-8881-0dd30efd2d0c" containerName="istio-proxy" Nov 03 23:07:51.321330 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.321302 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e36eab-2abd-4a73-b86d-be125deaa935" containerName="revoke-keys" Nov 03 23:07:51.321330 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.321314 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fac85f-2514-47c1-931a-37ccc7b40a6e" containerName="maas-api" Nov 03 23:07:51.326546 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.326522 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.348519 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.348491 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9589f759-tfgvc"] Nov 03 23:07:51.355919 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.355887 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 23:07:51.356135 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.355955 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rnrkl\"" Nov 03 23:07:51.356135 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.355977 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 23:07:51.356336 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.356033 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 23:07:51.363639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.363616 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-metrics-certs\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.363801 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.363658 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xdz\" (UniqueName: \"kubernetes.io/projected/0d2aadac-cac1-4116-9ab5-e888dd6dc792-kube-api-access-v6xdz\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.363801 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.363677 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-webhook-cert\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.464917 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.464880 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-metrics-certs\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.465134 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.464927 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xdz\" (UniqueName: \"kubernetes.io/projected/0d2aadac-cac1-4116-9ab5-e888dd6dc792-kube-api-access-v6xdz\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.465134 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.464950 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-webhook-cert\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.467487 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.467455 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-metrics-certs\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.467648 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.467625 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-webhook-cert\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.488085 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.488051 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xdz\" (UniqueName: \"kubernetes.io/projected/0d2aadac-cac1-4116-9ab5-e888dd6dc792-kube-api-access-v6xdz\") pod \"maas-controller-6d9589f759-tfgvc\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.641521 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.641479 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:51.951103 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.951010 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9589f759-tfgvc"] Nov 03 23:07:51.957499 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:51.957473 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:07:52.453153 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:52.453105 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-tfgvc" event={"ID":"0d2aadac-cac1-4116-9ab5-e888dd6dc792","Type":"ContainerStarted","Data":"65cec7e32a0bb6de6792456466a82aeed4fe0b9bd78b6d3b16480982b9b78e3f"} Nov 03 23:07:54.467892 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:54.467853 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-tfgvc" event={"ID":"0d2aadac-cac1-4116-9ab5-e888dd6dc792","Type":"ContainerStarted","Data":"344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852"} Nov 03 23:07:54.468323 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:54.468032 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:07:54.595567 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:07:54.595509 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9589f759-tfgvc" podStartSLOduration=1.968295401 podStartE2EDuration="3.595492145s" totalImagesPullingTime="1.627196744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:54.593690203 +0000 UTC m=+3296.023026196" watchObservedRunningTime="2026-11-03 23:07:54.595492145 +0000 UTC m=+3296.024828140" Nov 03 23:08:02.184670 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:08:02.184629 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:08:03.964918 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:03.964890 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:08:04.013187 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:04.013161 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:08:05.477785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:05.477757 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:08:07.554334 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:07.554296 4625 scope.go:122] "RemoveContainer" containerID="34e7ed06c0529e1d5ae3a07264addb49ba2995187df83dce301d070157438eef" Nov 03 23:08:12.110606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.110571 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc"] Nov 03 23:08:12.115602 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.115578 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.142677 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.142463 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:08:12.142877 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.142701 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:08:12.145343 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.145319 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc"] Nov 03 23:08:12.264245 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.264203 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb7923-7273-4062-a923-92478b8853f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.264448 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.264360 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb7923-7273-4062-a923-92478b8853f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.264535 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.264487 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfjb\" (UniqueName: \"kubernetes.io/projected/16bb7923-7273-4062-a923-92478b8853f8-kube-api-access-cxfjb\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.366067 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.365976 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb7923-7273-4062-a923-92478b8853f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.366067 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.366052 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfjb\" (UniqueName: \"kubernetes.io/projected/16bb7923-7273-4062-a923-92478b8853f8-kube-api-access-cxfjb\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.366304 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.366113 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb7923-7273-4062-a923-92478b8853f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.366786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.366751 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb7923-7273-4062-a923-92478b8853f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.368653 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.368625 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb7923-7273-4062-a923-92478b8853f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.388424 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.388398 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfjb\" (UniqueName: \"kubernetes.io/projected/16bb7923-7273-4062-a923-92478b8853f8-kube-api-access-cxfjb\") pod \"maas-api-revoke-sub-85237e45-q4xhc\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.432011 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.431982 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:12.744544 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:12.744513 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc"] Nov 03 23:08:12.746811 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:08:12.746780 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bb7923_7273_4062_a923_92478b8853f8.slice/crio-653fadcee89aafb37da8b70317c516fd062f03b235879edd077f9a539bff65d2 WatchSource:0}: Error finding container 653fadcee89aafb37da8b70317c516fd062f03b235879edd077f9a539bff65d2: Status 404 returned error can't find the container with id 653fadcee89aafb37da8b70317c516fd062f03b235879edd077f9a539bff65d2 Nov 03 23:08:13.581907 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:13.581876 4625 generic.go:356] "Generic (PLEG): container finished" podID="16bb7923-7273-4062-a923-92478b8853f8" containerID="2b71e5b296913984d36480c8d0c6eb4e8db5a02c592bf4cf0546032ba853700e" exitCode=0 Nov 03 23:08:13.582258 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:13.581963 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" event={"ID":"16bb7923-7273-4062-a923-92478b8853f8","Type":"ContainerDied","Data":"2b71e5b296913984d36480c8d0c6eb4e8db5a02c592bf4cf0546032ba853700e"} Nov 03 23:08:13.582258 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:13.582037 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" event={"ID":"16bb7923-7273-4062-a923-92478b8853f8","Type":"ContainerStarted","Data":"653fadcee89aafb37da8b70317c516fd062f03b235879edd077f9a539bff65d2"} Nov 03 23:08:14.728731 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.728700 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:14.790161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.790133 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16bb7923-7273-4062-a923-92478b8853f8-kube-api-access-cxfjb\" (UniqueName: \"kubernetes.io/projected/16bb7923-7273-4062-a923-92478b8853f8-kube-api-access-cxfjb\") pod \"16bb7923-7273-4062-a923-92478b8853f8\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " Nov 03 23:08:14.790360 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.790194 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16bb7923-7273-4062-a923-92478b8853f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb7923-7273-4062-a923-92478b8853f8-maas-api-xks-ca\") pod \"16bb7923-7273-4062-a923-92478b8853f8\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " Nov 03 23:08:14.790360 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.790253 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16bb7923-7273-4062-a923-92478b8853f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb7923-7273-4062-a923-92478b8853f8-maas-api-openshift-ca\") pod \"16bb7923-7273-4062-a923-92478b8853f8\" (UID: \"16bb7923-7273-4062-a923-92478b8853f8\") " Nov 03 23:08:14.790754 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.790695 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bb7923-7273-4062-a923-92478b8853f8-maas-api-openshift-ca" pod "16bb7923-7273-4062-a923-92478b8853f8" (UID: "16bb7923-7273-4062-a923-92478b8853f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:14.792936 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.792908 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bb7923-7273-4062-a923-92478b8853f8-maas-api-xks-ca" pod "16bb7923-7273-4062-a923-92478b8853f8" (UID: "16bb7923-7273-4062-a923-92478b8853f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:14.793036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.792940 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bb7923-7273-4062-a923-92478b8853f8-kube-api-access-cxfjb" pod "16bb7923-7273-4062-a923-92478b8853f8" (UID: "16bb7923-7273-4062-a923-92478b8853f8"). InnerVolumeSpecName "kube-api-access-cxfjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:14.891974 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.891899 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb7923-7273-4062-a923-92478b8853f8-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:08:14.891974 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.891925 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfjb\" (UniqueName: \"kubernetes.io/projected/16bb7923-7273-4062-a923-92478b8853f8-kube-api-access-cxfjb\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:08:14.891974 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:14.891937 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb7923-7273-4062-a923-92478b8853f8-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:08:15.595753 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:15.595702 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" Nov 03 23:08:15.595753 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:15.595741 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc" event={"ID":"16bb7923-7273-4062-a923-92478b8853f8","Type":"ContainerDied","Data":"653fadcee89aafb37da8b70317c516fd062f03b235879edd077f9a539bff65d2"} Nov 03 23:08:15.595973 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:15.595798 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="653fadcee89aafb37da8b70317c516fd062f03b235879edd077f9a539bff65d2" Nov 03 23:08:23.897323 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:23.897288 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9589f759-tfgvc"] Nov 03 23:08:23.898295 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:23.897559 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6d9589f759-tfgvc" podUID="0d2aadac-cac1-4116-9ab5-e888dd6dc792" containerName="manager" containerID="cri-o://344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852" gracePeriod=30 Nov 03 23:08:24.151594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.151529 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:08:24.282096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.282060 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-metrics-certs\") pod \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " Nov 03 23:08:24.282293 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.282125 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-webhook-cert\") pod \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " Nov 03 23:08:24.282293 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.282172 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2aadac-cac1-4116-9ab5-e888dd6dc792-kube-api-access-v6xdz\" (UniqueName: \"kubernetes.io/projected/0d2aadac-cac1-4116-9ab5-e888dd6dc792-kube-api-access-v6xdz\") pod \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\" (UID: \"0d2aadac-cac1-4116-9ab5-e888dd6dc792\") " Nov 03 23:08:24.284227 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.284200 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2aadac-cac1-4116-9ab5-e888dd6dc792-kube-api-access-v6xdz" pod "0d2aadac-cac1-4116-9ab5-e888dd6dc792" (UID: "0d2aadac-cac1-4116-9ab5-e888dd6dc792"). InnerVolumeSpecName "kube-api-access-v6xdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:24.284385 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.284245 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-webhook-cert" pod "0d2aadac-cac1-4116-9ab5-e888dd6dc792" (UID: "0d2aadac-cac1-4116-9ab5-e888dd6dc792"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:24.284385 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.284314 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-metrics-certs" pod "0d2aadac-cac1-4116-9ab5-e888dd6dc792" (UID: "0d2aadac-cac1-4116-9ab5-e888dd6dc792"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:24.383372 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.383333 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:08:24.383372 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.383364 4625 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d2aadac-cac1-4116-9ab5-e888dd6dc792-webhook-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:08:24.383372 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.383376 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xdz\" (UniqueName: \"kubernetes.io/projected/0d2aadac-cac1-4116-9ab5-e888dd6dc792-kube-api-access-v6xdz\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:08:24.646877 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.646841 4625 generic.go:356] "Generic (PLEG): container finished" podID="0d2aadac-cac1-4116-9ab5-e888dd6dc792" containerID="344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852" exitCode=0 Nov 03 23:08:24.647091 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.646890 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-tfgvc" Nov 03 23:08:24.647091 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.646923 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-tfgvc" event={"ID":"0d2aadac-cac1-4116-9ab5-e888dd6dc792","Type":"ContainerDied","Data":"344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852"} Nov 03 23:08:24.647091 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.646982 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-tfgvc" event={"ID":"0d2aadac-cac1-4116-9ab5-e888dd6dc792","Type":"ContainerDied","Data":"65cec7e32a0bb6de6792456466a82aeed4fe0b9bd78b6d3b16480982b9b78e3f"} Nov 03 23:08:24.647091 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.647007 4625 scope.go:122] "RemoveContainer" containerID="344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852" Nov 03 23:08:24.656421 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.656404 4625 scope.go:122] "RemoveContainer" containerID="344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852" Nov 03 23:08:24.656750 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:08:24.656702 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852\": container with ID starting with 344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852 not found: ID does not exist" containerID="344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852" Nov 03 23:08:24.656850 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.656758 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852"} err="failed to get container status \"344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852\": rpc error: code = NotFound desc = could not find container \"344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852\": container with ID starting with 344af6214e8b7a63140d1a17e8c13013550f8c82b6eb5575931de098c5fb6852 not found: ID does not exist" Nov 03 23:08:24.695778 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.695740 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9589f759-tfgvc"] Nov 03 23:08:24.706985 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.706950 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d9589f759-tfgvc"] Nov 03 23:08:24.726428 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:24.726401 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2aadac-cac1-4116-9ab5-e888dd6dc792" path="/var/lib/kubelet/pods/0d2aadac-cac1-4116-9ab5-e888dd6dc792/volumes" Nov 03 23:08:25.868332 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.868298 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9589f759-xg97m"] Nov 03 23:08:25.868771 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.868736 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0d2aadac-cac1-4116-9ab5-e888dd6dc792" containerName="manager" Nov 03 23:08:25.868771 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.868749 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2aadac-cac1-4116-9ab5-e888dd6dc792" containerName="manager" Nov 03 23:08:25.868771 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.868759 4625 cpu_manager.go:414] "containerMap: removing container" podUID="16bb7923-7273-4062-a923-92478b8853f8" containerName="revoke-keys" Nov 03 23:08:25.868771 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.868767 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16bb7923-7273-4062-a923-92478b8853f8" containerName="revoke-keys" Nov 03 23:08:25.868923 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.868834 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2aadac-cac1-4116-9ab5-e888dd6dc792" containerName="manager" Nov 03 23:08:25.868923 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.868844 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="16bb7923-7273-4062-a923-92478b8853f8" containerName="revoke-keys" Nov 03 23:08:25.873420 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.873399 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:25.901832 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.901804 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 23:08:25.901971 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.901852 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rnrkl\"" Nov 03 23:08:25.901971 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.901877 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 23:08:25.901971 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.901887 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 23:08:25.916829 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.916803 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9589f759-xg97m"] Nov 03 23:08:25.998241 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.998207 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91811bd7-2f89-4a68-8263-7d231eadc5b7-metrics-certs\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:25.998437 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.998266 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7nt\" (UniqueName: \"kubernetes.io/projected/91811bd7-2f89-4a68-8263-7d231eadc5b7-kube-api-access-9s7nt\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:25.998437 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:25.998367 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91811bd7-2f89-4a68-8263-7d231eadc5b7-webhook-cert\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.099366 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.099330 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91811bd7-2f89-4a68-8263-7d231eadc5b7-metrics-certs\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.099558 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.099400 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7nt\" (UniqueName: \"kubernetes.io/projected/91811bd7-2f89-4a68-8263-7d231eadc5b7-kube-api-access-9s7nt\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.099558 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.099439 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91811bd7-2f89-4a68-8263-7d231eadc5b7-webhook-cert\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.102230 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.102197 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91811bd7-2f89-4a68-8263-7d231eadc5b7-webhook-cert\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.102387 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.102197 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91811bd7-2f89-4a68-8263-7d231eadc5b7-metrics-certs\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.122257 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.122176 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7nt\" (UniqueName: \"kubernetes.io/projected/91811bd7-2f89-4a68-8263-7d231eadc5b7-kube-api-access-9s7nt\") pod \"maas-controller-6d9589f759-xg97m\" (UID: \"91811bd7-2f89-4a68-8263-7d231eadc5b7\") " pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.186940 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.186899 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:26.491600 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.491573 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9589f759-xg97m"] Nov 03 23:08:26.493838 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:08:26.493810 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91811bd7_2f89_4a68_8263_7d231eadc5b7.slice/crio-c59add32fdd71e06f3506d62ecf895cdb06ca7629d2bf9e7c23d8958c75fac7d WatchSource:0}: Error finding container c59add32fdd71e06f3506d62ecf895cdb06ca7629d2bf9e7c23d8958c75fac7d: Status 404 returned error can't find the container with id c59add32fdd71e06f3506d62ecf895cdb06ca7629d2bf9e7c23d8958c75fac7d Nov 03 23:08:26.660772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:26.660736 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-xg97m" event={"ID":"91811bd7-2f89-4a68-8263-7d231eadc5b7","Type":"ContainerStarted","Data":"c59add32fdd71e06f3506d62ecf895cdb06ca7629d2bf9e7c23d8958c75fac7d"} Nov 03 23:08:27.668284 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:27.668234 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-xg97m" event={"ID":"91811bd7-2f89-4a68-8263-7d231eadc5b7","Type":"ContainerStarted","Data":"7cfe25fa6065bc56011855cb7ab4b1c2d5c9182e184e0c0b4dad8fb8757411d0"} Nov 03 23:08:27.668735 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:27.668444 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:08:27.712061 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:27.712001 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9589f759-xg97m" podStartSLOduration=2.379375055 podStartE2EDuration="2.711983982s" totalImagesPullingTime="332.608927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:27.709940569 +0000 UTC m=+3329.139276562" watchObservedRunningTime="2026-11-03 23:08:27.711983982 +0000 UTC m=+3329.141319975" Nov 03 23:08:32.185783 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:08:32.185750 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:08:38.677119 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:08:38.677087 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9589f759-xg97m" Nov 03 23:09:02.184508 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:09:02.184467 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:09:11.038365 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.038259 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696"] Nov 03 23:09:11.042996 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.042965 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.060767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.060734 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696"] Nov 03 23:09:11.069469 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.068835 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:09:11.069469 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.068889 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:09:11.236863 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.236831 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvnp\" (UniqueName: \"kubernetes.io/projected/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-kube-api-access-dcvnp\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.237062 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.236872 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.237062 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.236994 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.337959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.337881 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvnp\" (UniqueName: \"kubernetes.io/projected/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-kube-api-access-dcvnp\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.337959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.337922 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.338145 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.337979 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.338603 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.338574 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.340245 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.340221 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.360542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.360514 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvnp\" (UniqueName: \"kubernetes.io/projected/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-kube-api-access-dcvnp\") pod \"maas-api-revoke-sub-c1e271a9-sj696\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.656476 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.656437 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:11.962664 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:11.962471 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696"] Nov 03 23:09:11.965748 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:09:11.965698 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4330f85_82ce_45df_a0b3_ae1e3feee0c1.slice/crio-b35162eab4d7640469e5e162664d44ba42cbe723094237d1d17fd04a3726bf0f WatchSource:0}: Error finding container b35162eab4d7640469e5e162664d44ba42cbe723094237d1d17fd04a3726bf0f: Status 404 returned error can't find the container with id b35162eab4d7640469e5e162664d44ba42cbe723094237d1d17fd04a3726bf0f Nov 03 23:09:12.936622 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:12.936588 4625 generic.go:356] "Generic (PLEG): container finished" podID="f4330f85-82ce-45df-a0b3-ae1e3feee0c1" containerID="d97bb797ef7c8bd355ebd90192a21dd82fbdae9c442040dd4637444748753f0b" exitCode=0 Nov 03 23:09:12.937087 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:12.936674 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" event={"ID":"f4330f85-82ce-45df-a0b3-ae1e3feee0c1","Type":"ContainerDied","Data":"d97bb797ef7c8bd355ebd90192a21dd82fbdae9c442040dd4637444748753f0b"} Nov 03 23:09:12.937087 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:12.936762 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" event={"ID":"f4330f85-82ce-45df-a0b3-ae1e3feee0c1","Type":"ContainerStarted","Data":"b35162eab4d7640469e5e162664d44ba42cbe723094237d1d17fd04a3726bf0f"} Nov 03 23:09:14.087235 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.087212 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:14.163409 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.163374 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-openshift-ca\") pod \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " Nov 03 23:09:14.163594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.163434 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-xks-ca\") pod \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " Nov 03 23:09:14.163594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.163487 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-kube-api-access-dcvnp\" (UniqueName: \"kubernetes.io/projected/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-kube-api-access-dcvnp\") pod \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\" (UID: \"f4330f85-82ce-45df-a0b3-ae1e3feee0c1\") " Nov 03 23:09:14.163859 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.163831 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-openshift-ca" pod "f4330f85-82ce-45df-a0b3-ae1e3feee0c1" (UID: "f4330f85-82ce-45df-a0b3-ae1e3feee0c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:14.165509 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.165484 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-xks-ca" pod "f4330f85-82ce-45df-a0b3-ae1e3feee0c1" (UID: "f4330f85-82ce-45df-a0b3-ae1e3feee0c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:14.165693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.165673 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-kube-api-access-dcvnp" pod "f4330f85-82ce-45df-a0b3-ae1e3feee0c1" (UID: "f4330f85-82ce-45df-a0b3-ae1e3feee0c1"). InnerVolumeSpecName "kube-api-access-dcvnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:14.264693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.264608 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:14.264693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.264637 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:14.264693 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.264650 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvnp\" (UniqueName: \"kubernetes.io/projected/f4330f85-82ce-45df-a0b3-ae1e3feee0c1-kube-api-access-dcvnp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:14.956837 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.956802 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" Nov 03 23:09:14.957024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.956828 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696" event={"ID":"f4330f85-82ce-45df-a0b3-ae1e3feee0c1","Type":"ContainerDied","Data":"b35162eab4d7640469e5e162664d44ba42cbe723094237d1d17fd04a3726bf0f"} Nov 03 23:09:14.957024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:14.956890 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b35162eab4d7640469e5e162664d44ba42cbe723094237d1d17fd04a3726bf0f" Nov 03 23:09:31.039654 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.039615 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg"] Nov 03 23:09:31.040070 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.040049 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f4330f85-82ce-45df-a0b3-ae1e3feee0c1" containerName="revoke-keys" Nov 03 23:09:31.040070 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.040062 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4330f85-82ce-45df-a0b3-ae1e3feee0c1" containerName="revoke-keys" Nov 03 23:09:31.040148 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.040141 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4330f85-82ce-45df-a0b3-ae1e3feee0c1" containerName="revoke-keys" Nov 03 23:09:31.043441 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.043421 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.059744 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.059696 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:09:31.070872 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.070837 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:09:31.074256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.074229 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg"] Nov 03 23:09:31.126686 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.126656 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.126906 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.126704 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.126906 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.126842 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jmp\" (UniqueName: \"kubernetes.io/projected/43060d43-6985-4fbe-a15a-c4122054796d-kube-api-access-b7jmp\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.227542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.227504 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.227780 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.227564 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jmp\" (UniqueName: \"kubernetes.io/projected/43060d43-6985-4fbe-a15a-c4122054796d-kube-api-access-b7jmp\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.227780 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.227677 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.228441 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.228407 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.230147 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.230124 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.249905 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.249874 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jmp\" (UniqueName: \"kubernetes.io/projected/43060d43-6985-4fbe-a15a-c4122054796d-kube-api-access-b7jmp\") pod \"maas-api-revoke-sub-2e1f5b5f-x6lqg\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.357289 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.357256 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:31.660283 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:31.660141 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg"] Nov 03 23:09:31.662710 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:09:31.662682 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43060d43_6985_4fbe_a15a_c4122054796d.slice/crio-53060c0154184da6bcb4239cc2b6fa4c444b732e7aac1bc1309008a3bebc7971 WatchSource:0}: Error finding container 53060c0154184da6bcb4239cc2b6fa4c444b732e7aac1bc1309008a3bebc7971: Status 404 returned error can't find the container with id 53060c0154184da6bcb4239cc2b6fa4c444b732e7aac1bc1309008a3bebc7971 Nov 03 23:09:32.066434 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:32.066332 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" event={"ID":"43060d43-6985-4fbe-a15a-c4122054796d","Type":"ContainerStarted","Data":"468a5fa4e1fbb30dad882a589c61b3ad6029b804024a785dbed806a32e90c108"} Nov 03 23:09:32.066434 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:32.066392 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" event={"ID":"43060d43-6985-4fbe-a15a-c4122054796d","Type":"ContainerStarted","Data":"53060c0154184da6bcb4239cc2b6fa4c444b732e7aac1bc1309008a3bebc7971"} Nov 03 23:09:32.107660 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:32.107602 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" podStartSLOduration=1.107585344 podStartE2EDuration="1.107585344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:32.103769359 +0000 UTC m=+3393.533105352" watchObservedRunningTime="2026-11-03 23:09:32.107585344 +0000 UTC m=+3393.536921337" Nov 03 23:09:32.185162 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:09:32.185118 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:09:33.074023 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:33.073991 4625 generic.go:356] "Generic (PLEG): container finished" podID="43060d43-6985-4fbe-a15a-c4122054796d" containerID="468a5fa4e1fbb30dad882a589c61b3ad6029b804024a785dbed806a32e90c108" exitCode=0 Nov 03 23:09:33.074526 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:33.074085 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" event={"ID":"43060d43-6985-4fbe-a15a-c4122054796d","Type":"ContainerDied","Data":"468a5fa4e1fbb30dad882a589c61b3ad6029b804024a785dbed806a32e90c108"} Nov 03 23:09:34.223881 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.223855 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:34.358564 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.358532 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-xks-ca\") pod \"43060d43-6985-4fbe-a15a-c4122054796d\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " Nov 03 23:09:34.358767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.358570 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43060d43-6985-4fbe-a15a-c4122054796d-kube-api-access-b7jmp\" (UniqueName: \"kubernetes.io/projected/43060d43-6985-4fbe-a15a-c4122054796d-kube-api-access-b7jmp\") pod \"43060d43-6985-4fbe-a15a-c4122054796d\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " Nov 03 23:09:34.358767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.358663 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-openshift-ca\") pod \"43060d43-6985-4fbe-a15a-c4122054796d\" (UID: \"43060d43-6985-4fbe-a15a-c4122054796d\") " Nov 03 23:09:34.359090 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.359053 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-openshift-ca" pod "43060d43-6985-4fbe-a15a-c4122054796d" (UID: "43060d43-6985-4fbe-a15a-c4122054796d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:34.360772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.360752 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43060d43-6985-4fbe-a15a-c4122054796d-kube-api-access-b7jmp" pod "43060d43-6985-4fbe-a15a-c4122054796d" (UID: "43060d43-6985-4fbe-a15a-c4122054796d"). InnerVolumeSpecName "kube-api-access-b7jmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:34.360842 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.360819 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-xks-ca" pod "43060d43-6985-4fbe-a15a-c4122054796d" (UID: "43060d43-6985-4fbe-a15a-c4122054796d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:34.459834 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.459800 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:34.459834 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.459827 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43060d43-6985-4fbe-a15a-c4122054796d-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:34.459834 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:34.459836 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jmp\" (UniqueName: \"kubernetes.io/projected/43060d43-6985-4fbe-a15a-c4122054796d-kube-api-access-b7jmp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:35.090034 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:35.090007 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" Nov 03 23:09:35.090206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:35.090039 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg" event={"ID":"43060d43-6985-4fbe-a15a-c4122054796d","Type":"ContainerDied","Data":"53060c0154184da6bcb4239cc2b6fa4c444b732e7aac1bc1309008a3bebc7971"} Nov 03 23:09:35.090206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:35.090082 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53060c0154184da6bcb4239cc2b6fa4c444b732e7aac1bc1309008a3bebc7971" Nov 03 23:09:38.224333 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.224291 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg"] Nov 03 23:09:38.224977 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.224954 4625 cpu_manager.go:414] "containerMap: removing container" podUID="43060d43-6985-4fbe-a15a-c4122054796d" containerName="revoke-keys" Nov 03 23:09:38.224977 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.224976 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43060d43-6985-4fbe-a15a-c4122054796d" containerName="revoke-keys" Nov 03 23:09:38.225131 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.225071 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="43060d43-6985-4fbe-a15a-c4122054796d" containerName="revoke-keys" Nov 03 23:09:38.228567 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.228549 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.257586 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.257534 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg"] Nov 03 23:09:38.298660 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.298620 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.298825 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.298672 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94vx\" (UniqueName: \"kubernetes.io/projected/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-kube-api-access-l94vx\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.298825 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.298701 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.298927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.298843 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.298927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.298885 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.298927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.298909 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.299087 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.299027 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.299128 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.299097 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.299180 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.299141 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.399821 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.399782 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94vx\" (UniqueName: \"kubernetes.io/projected/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-kube-api-access-l94vx\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.399997 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.399829 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.399997 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.399865 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.399997 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.399916 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400050 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400116 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400246 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400162 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400246 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400222 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400371 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400267 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400371 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400345 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400484 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400430 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400653 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400630 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400734 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400661 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.400734 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.400686 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.402404 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.402380 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.402696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.402677 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.422301 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.422267 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.422436 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.422354 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94vx\" (UniqueName: \"kubernetes.io/projected/13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc-kube-api-access-l94vx\") pod \"maas-default-gateway-openshift-default-7df854bf67-mfkdg\" (UID: \"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.543780 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.543670 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:38.866105 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.866080 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg"] Nov 03 23:09:38.868368 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:09:38.868333 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13369b2f_fa0e_42f6_8f6d_9d34c75ed1fc.slice/crio-50ee91c84d1d32636c76afc87da2f5d7ce9f4782188c757118e0cb15ec93e956 WatchSource:0}: Error finding container 50ee91c84d1d32636c76afc87da2f5d7ce9f4782188c757118e0cb15ec93e956: Status 404 returned error can't find the container with id 50ee91c84d1d32636c76afc87da2f5d7ce9f4782188c757118e0cb15ec93e956 Nov 03 23:09:38.870589 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.870558 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:09:38.870650 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.870625 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:09:38.870688 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:38.870655 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:09:39.115008 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:39.114969 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" event={"ID":"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc","Type":"ContainerStarted","Data":"2ebfdb770fb273d8689707ac1cf9abc4071d2aa0515d5f8006a3de347ee89245"} Nov 03 23:09:39.115214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:39.115026 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" event={"ID":"13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc","Type":"ContainerStarted","Data":"50ee91c84d1d32636c76afc87da2f5d7ce9f4782188c757118e0cb15ec93e956"} Nov 03 23:09:39.159708 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:39.159599 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podStartSLOduration=1.159582144 podStartE2EDuration="1.159582144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:39.157108768 +0000 UTC m=+3400.586444760" watchObservedRunningTime="2026-11-03 23:09:39.159582144 +0000 UTC m=+3400.588918136" Nov 03 23:09:39.547798 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:39.546215 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:39.547798 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:39.546367 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 23:09:39.547798 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:39.546409 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 23:09:40.554741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:40.548110 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 23:09:40.554741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:40.548168 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 23:09:41.553741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:41.550054 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 23:09:41.553741 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:41.550114 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 23:09:42.362266 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.362225 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd"] Nov 03 23:09:42.366064 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.366040 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.391990 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.391962 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:09:42.392219 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.392026 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:09:42.395967 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.395933 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd"] Nov 03 23:09:42.471070 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.471033 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8xp\" (UniqueName: \"kubernetes.io/projected/4a627397-085a-4dfe-8474-be8a251076e0-kube-api-access-lj8xp\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.471070 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.471068 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.471332 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.471187 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.551164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.551124 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 23:09:42.551470 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.551438 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 23:09:42.572159 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.572124 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8xp\" (UniqueName: \"kubernetes.io/projected/4a627397-085a-4dfe-8474-be8a251076e0-kube-api-access-lj8xp\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.572159 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.572162 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.572443 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.572206 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.573075 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.573045 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.574669 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.574639 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.593947 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.593919 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8xp\" (UniqueName: \"kubernetes.io/projected/4a627397-085a-4dfe-8474-be8a251076e0-kube-api-access-lj8xp\") pod \"maas-api-revoke-sub-ec843018-zvzmd\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:42.680215 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:42.680135 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:43.014492 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:43.014438 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd"] Nov 03 23:09:43.017052 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:09:43.017021 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a627397_085a_4dfe_8474_be8a251076e0.slice/crio-017839bdbe779529d6ba80026a05a742f4a512332c419d923bb964f4ab600798 WatchSource:0}: Error finding container 017839bdbe779529d6ba80026a05a742f4a512332c419d923bb964f4ab600798: Status 404 returned error can't find the container with id 017839bdbe779529d6ba80026a05a742f4a512332c419d923bb964f4ab600798 Nov 03 23:09:43.162182 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:43.162123 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" event={"ID":"4a627397-085a-4dfe-8474-be8a251076e0","Type":"ContainerStarted","Data":"017839bdbe779529d6ba80026a05a742f4a512332c419d923bb964f4ab600798"} Nov 03 23:09:43.544959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:43.544867 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 23:09:43.545505 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:43.545475 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 23:09:44.170409 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:44.170367 4625 generic.go:356] "Generic (PLEG): container finished" podID="4a627397-085a-4dfe-8474-be8a251076e0" containerID="d0c5670ae13084b48d0a306c7e5abdb3009f20f42439e4afb897fcfdee0d0c5f" exitCode=0 Nov 03 23:09:44.170622 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:44.170460 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" event={"ID":"4a627397-085a-4dfe-8474-be8a251076e0","Type":"ContainerDied","Data":"d0c5670ae13084b48d0a306c7e5abdb3009f20f42439e4afb897fcfdee0d0c5f"} Nov 03 23:09:44.553735 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:44.546072 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 23:09:44.553735 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:44.546133 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 23:09:45.329160 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.329131 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:45.406581 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.406536 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a627397-085a-4dfe-8474-be8a251076e0-kube-api-access-lj8xp\" (UniqueName: \"kubernetes.io/projected/4a627397-085a-4dfe-8474-be8a251076e0-kube-api-access-lj8xp\") pod \"4a627397-085a-4dfe-8474-be8a251076e0\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " Nov 03 23:09:45.406803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.406632 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-openshift-ca\") pod \"4a627397-085a-4dfe-8474-be8a251076e0\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " Nov 03 23:09:45.406803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.406666 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-xks-ca\") pod \"4a627397-085a-4dfe-8474-be8a251076e0\" (UID: \"4a627397-085a-4dfe-8474-be8a251076e0\") " Nov 03 23:09:45.407114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.407081 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-openshift-ca" pod "4a627397-085a-4dfe-8474-be8a251076e0" (UID: "4a627397-085a-4dfe-8474-be8a251076e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:45.408795 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.408769 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a627397-085a-4dfe-8474-be8a251076e0-kube-api-access-lj8xp" pod "4a627397-085a-4dfe-8474-be8a251076e0" (UID: "4a627397-085a-4dfe-8474-be8a251076e0"). InnerVolumeSpecName "kube-api-access-lj8xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:45.408931 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.408842 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-xks-ca" pod "4a627397-085a-4dfe-8474-be8a251076e0" (UID: "4a627397-085a-4dfe-8474-be8a251076e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:45.508259 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.508165 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:45.508259 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.508195 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a627397-085a-4dfe-8474-be8a251076e0-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:45.508259 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.508205 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8xp\" (UniqueName: \"kubernetes.io/projected/4a627397-085a-4dfe-8474-be8a251076e0-kube-api-access-lj8xp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:09:45.543989 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.543943 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 23:09:45.544201 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:45.543999 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 23:09:46.186973 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:46.186935 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" event={"ID":"4a627397-085a-4dfe-8474-be8a251076e0","Type":"ContainerDied","Data":"017839bdbe779529d6ba80026a05a742f4a512332c419d923bb964f4ab600798"} Nov 03 23:09:46.187505 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:46.186985 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="017839bdbe779529d6ba80026a05a742f4a512332c419d923bb964f4ab600798" Nov 03 23:09:46.187505 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:46.186951 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd" Nov 03 23:09:47.545712 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:47.545670 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:09:47.546221 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:47.545741 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:48.586900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:48.586842 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:09:48.587413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:48.586918 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:49.242264 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:49.242226 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:49.242578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:49.242550 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:49.243314 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:49.243293 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" Nov 03 23:09:53.247117 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.247080 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2"] Nov 03 23:09:53.247732 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.247695 4625 cpu_manager.go:414] "containerMap: removing container" podUID="4a627397-085a-4dfe-8474-be8a251076e0" containerName="revoke-keys" Nov 03 23:09:53.247732 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.247730 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a627397-085a-4dfe-8474-be8a251076e0" containerName="revoke-keys" Nov 03 23:09:53.247924 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.247899 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a627397-085a-4dfe-8474-be8a251076e0" containerName="revoke-keys" Nov 03 23:09:53.250961 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.250933 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.280911 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.280879 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2"] Nov 03 23:09:53.285271 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285245 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v45t\" (UniqueName: \"kubernetes.io/projected/e6eb71bf-3a09-4a93-b518-91961518a97e-kube-api-access-6v45t\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285407 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285293 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285407 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285312 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285407 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285354 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285407 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285372 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285407 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285387 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285415 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285465 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.285639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.285524 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6eb71bf-3a09-4a93-b518-91961518a97e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386386 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386347 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386386 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386387 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386409 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386445 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386500 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386545 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6eb71bf-3a09-4a93-b518-91961518a97e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386604 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v45t\" (UniqueName: \"kubernetes.io/projected/e6eb71bf-3a09-4a93-b518-91961518a97e-kube-api-access-6v45t\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.386670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386661 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.387030 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386689 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.387030 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.386952 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.387300 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.387269 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.387433 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.387397 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.387576 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.387553 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.387623 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.387554 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6eb71bf-3a09-4a93-b518-91961518a97e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.389236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.389206 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.389418 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.389400 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.411614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.411593 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6eb71bf-3a09-4a93-b518-91961518a97e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.411863 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.411841 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v45t\" (UniqueName: \"kubernetes.io/projected/e6eb71bf-3a09-4a93-b518-91961518a97e-kube-api-access-6v45t\") pod \"maas-default-gateway-openshift-default-7df854bf67-gprr2\" (UID: \"e6eb71bf-3a09-4a93-b518-91961518a97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.564841 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.564746 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:53.888692 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.888671 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2"] Nov 03 23:09:53.890780 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:09:53.890743 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6eb71bf_3a09_4a93_b518_91961518a97e.slice/crio-f902ab9612d561ed2bceb8e84d66af72d0ef21a9e3697e95cdf2e093acefc93e WatchSource:0}: Error finding container f902ab9612d561ed2bceb8e84d66af72d0ef21a9e3697e95cdf2e093acefc93e: Status 404 returned error can't find the container with id f902ab9612d561ed2bceb8e84d66af72d0ef21a9e3697e95cdf2e093acefc93e Nov 03 23:09:53.893039 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.893006 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:09:53.893115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.893069 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:09:53.893115 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:53.893098 4625 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 23:09:54.244304 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:54.244195 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" event={"ID":"e6eb71bf-3a09-4a93-b518-91961518a97e","Type":"ContainerStarted","Data":"da22ea5480cc614d1e08de2c2ba9eb76a0d167eea9960e2885f80fb918467056"} Nov 03 23:09:54.244578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:54.244548 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" event={"ID":"e6eb71bf-3a09-4a93-b518-91961518a97e","Type":"ContainerStarted","Data":"f902ab9612d561ed2bceb8e84d66af72d0ef21a9e3697e95cdf2e093acefc93e"} Nov 03 23:09:54.313310 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:54.313244 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podStartSLOduration=1.3132271150000001 podStartE2EDuration="1.313227115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:54.299307191 +0000 UTC m=+3415.728643185" watchObservedRunningTime="2026-11-03 23:09:54.313227115 +0000 UTC m=+3415.742563107" Nov 03 23:09:54.565399 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:54.565309 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:09:54.567093 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:54.567063 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:09:54.567229 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:54.567114 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:09:55.565429 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:55.565396 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:09:55.565890 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:55.565449 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:09:56.565069 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:56.565033 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:09:56.565288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:56.565098 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:09:57.565046 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:57.565015 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:09:57.565569 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:57.565279 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:09:58.565153 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:58.565112 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:09:58.565588 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:58.565170 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:09:59.568880 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:59.568835 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:09:59.569449 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:09:59.568902 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:10:00.571742 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:00.570778 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:10:00.571742 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:00.570843 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:10:01.567761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:01.567694 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 23:10:01.567991 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:01.567769 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 23:10:02.189708 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:10:02.189666 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:10:03.605878 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:03.605841 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:10:03.606428 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:03.605909 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:10:04.647851 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:04.647814 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:10:04.650540 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:04.647865 4625 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:10:04.771030 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:04.770991 4625 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:10:04.771392 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:04.771373 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:10:04.772140 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:04.772118 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" Nov 03 23:10:10.378966 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.378932 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp"] Nov 03 23:10:10.382975 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.382954 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.407121 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.407095 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp"] Nov 03 23:10:10.407877 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.407850 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.408003 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.407891 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvspp\" (UniqueName: \"kubernetes.io/projected/e2f4a165-bc87-4ed1-9d44-920f128d4a77-kube-api-access-nvspp\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.408099 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.408073 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.414808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.414785 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:10:10.414808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.414785 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:10.508975 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.508943 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.508975 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.508983 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvspp\" (UniqueName: \"kubernetes.io/projected/e2f4a165-bc87-4ed1-9d44-920f128d4a77-kube-api-access-nvspp\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.509263 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.509056 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.509689 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.509661 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.511420 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.511398 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.531273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.531236 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvspp\" (UniqueName: \"kubernetes.io/projected/e2f4a165-bc87-4ed1-9d44-920f128d4a77-kube-api-access-nvspp\") pod \"maas-api-revoke-sub-046e1343-qfrfp\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:10.696454 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:10.696368 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:11.001983 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:11.001957 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp"] Nov 03 23:10:11.003807 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:10:11.003775 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f4a165_bc87_4ed1_9d44_920f128d4a77.slice/crio-336a6c53ec8483140aa04c637f5a6e07f7a56fcacfac302bce58c99cb811fe44 WatchSource:0}: Error finding container 336a6c53ec8483140aa04c637f5a6e07f7a56fcacfac302bce58c99cb811fe44: Status 404 returned error can't find the container with id 336a6c53ec8483140aa04c637f5a6e07f7a56fcacfac302bce58c99cb811fe44 Nov 03 23:10:11.370477 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:11.370445 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" event={"ID":"e2f4a165-bc87-4ed1-9d44-920f128d4a77","Type":"ContainerStarted","Data":"37c0a6abbc1f0e3793c4ae504845646b2bbcdf8a91f330312d02b4531f0294d7"} Nov 03 23:10:11.370621 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:11.370505 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" event={"ID":"e2f4a165-bc87-4ed1-9d44-920f128d4a77","Type":"ContainerStarted","Data":"336a6c53ec8483140aa04c637f5a6e07f7a56fcacfac302bce58c99cb811fe44"} Nov 03 23:10:11.409003 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:11.408951 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" podStartSLOduration=1.408936591 podStartE2EDuration="1.408936591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:10:11.407984897 +0000 UTC m=+3432.837320890" watchObservedRunningTime="2026-11-03 23:10:11.408936591 +0000 UTC m=+3432.838272654" Nov 03 23:10:12.377906 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:12.377873 4625 generic.go:356] "Generic (PLEG): container finished" podID="e2f4a165-bc87-4ed1-9d44-920f128d4a77" containerID="37c0a6abbc1f0e3793c4ae504845646b2bbcdf8a91f330312d02b4531f0294d7" exitCode=0 Nov 03 23:10:12.378113 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:12.377963 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" event={"ID":"e2f4a165-bc87-4ed1-9d44-920f128d4a77","Type":"ContainerDied","Data":"37c0a6abbc1f0e3793c4ae504845646b2bbcdf8a91f330312d02b4531f0294d7"} Nov 03 23:10:13.517681 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.517650 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:13.536033 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.536010 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f4a165-bc87-4ed1-9d44-920f128d4a77-kube-api-access-nvspp\" (UniqueName: \"kubernetes.io/projected/e2f4a165-bc87-4ed1-9d44-920f128d4a77-kube-api-access-nvspp\") pod \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " Nov 03 23:10:13.536151 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.536041 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-openshift-ca\") pod \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " Nov 03 23:10:13.536151 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.536121 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-xks-ca\") pod \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\" (UID: \"e2f4a165-bc87-4ed1-9d44-920f128d4a77\") " Nov 03 23:10:13.536406 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.536383 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-openshift-ca" pod "e2f4a165-bc87-4ed1-9d44-920f128d4a77" (UID: "e2f4a165-bc87-4ed1-9d44-920f128d4a77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:13.538078 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.538054 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f4a165-bc87-4ed1-9d44-920f128d4a77-kube-api-access-nvspp" pod "e2f4a165-bc87-4ed1-9d44-920f128d4a77" (UID: "e2f4a165-bc87-4ed1-9d44-920f128d4a77"). InnerVolumeSpecName "kube-api-access-nvspp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:13.538172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.538087 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-xks-ca" pod "e2f4a165-bc87-4ed1-9d44-920f128d4a77" (UID: "e2f4a165-bc87-4ed1-9d44-920f128d4a77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:13.636992 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.636915 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvspp\" (UniqueName: \"kubernetes.io/projected/e2f4a165-bc87-4ed1-9d44-920f128d4a77-kube-api-access-nvspp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:13.636992 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.636942 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:13.636992 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:13.636952 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f4a165-bc87-4ed1-9d44-920f128d4a77-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:14.390772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:14.390743 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" Nov 03 23:10:14.390772 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:14.390749 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp" event={"ID":"e2f4a165-bc87-4ed1-9d44-920f128d4a77","Type":"ContainerDied","Data":"336a6c53ec8483140aa04c637f5a6e07f7a56fcacfac302bce58c99cb811fe44"} Nov 03 23:10:14.390995 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:14.390794 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336a6c53ec8483140aa04c637f5a6e07f7a56fcacfac302bce58c99cb811fe44" Nov 03 23:10:20.806659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.806625 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp"] Nov 03 23:10:20.807063 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.807046 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e2f4a165-bc87-4ed1-9d44-920f128d4a77" containerName="revoke-keys" Nov 03 23:10:20.807063 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.807056 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f4a165-bc87-4ed1-9d44-920f128d4a77" containerName="revoke-keys" Nov 03 23:10:20.807150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.807143 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f4a165-bc87-4ed1-9d44-920f128d4a77" containerName="revoke-keys" Nov 03 23:10:20.810620 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.810599 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:20.826072 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.826045 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:20.837767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.837704 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:10:20.840781 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.840759 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp"] Nov 03 23:10:20.911047 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.911011 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:20.911244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.911056 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:20.911244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:20.911178 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wsgp\" (UniqueName: \"kubernetes.io/projected/b493fce4-3699-4660-b793-111e316e9cc1-kube-api-access-9wsgp\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.012348 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.012300 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.012577 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.012358 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.012577 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.012430 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wsgp\" (UniqueName: \"kubernetes.io/projected/b493fce4-3699-4660-b793-111e316e9cc1-kube-api-access-9wsgp\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.013150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.013127 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.014765 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.014735 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.034775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.034743 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wsgp\" (UniqueName: \"kubernetes.io/projected/b493fce4-3699-4660-b793-111e316e9cc1-kube-api-access-9wsgp\") pod \"maas-api-revoke-sub-d3fba056-58snp\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.123605 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.123575 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:21.427687 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:21.427662 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp"] Nov 03 23:10:21.430536 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:10:21.430501 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb493fce4_3699_4660_b793_111e316e9cc1.slice/crio-8b443141eedf27b219abdf1b245dbc471794e7bdf82e450411c38d671523bae0 WatchSource:0}: Error finding container 8b443141eedf27b219abdf1b245dbc471794e7bdf82e450411c38d671523bae0: Status 404 returned error can't find the container with id 8b443141eedf27b219abdf1b245dbc471794e7bdf82e450411c38d671523bae0 Nov 03 23:10:22.439256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:22.439221 4625 generic.go:356] "Generic (PLEG): container finished" podID="b493fce4-3699-4660-b793-111e316e9cc1" containerID="e74fab42cca482f036bb0a181517ff7f558c6412f9c05ef7623b8b3613fb7bff" exitCode=0 Nov 03 23:10:22.439781 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:22.439255 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" event={"ID":"b493fce4-3699-4660-b793-111e316e9cc1","Type":"ContainerDied","Data":"e74fab42cca482f036bb0a181517ff7f558c6412f9c05ef7623b8b3613fb7bff"} Nov 03 23:10:22.439905 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:22.439826 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" event={"ID":"b493fce4-3699-4660-b793-111e316e9cc1","Type":"ContainerStarted","Data":"8b443141eedf27b219abdf1b245dbc471794e7bdf82e450411c38d671523bae0"} Nov 03 23:10:24.116897 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.116864 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:24.229760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.229295 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-openshift-ca\") pod \"b493fce4-3699-4660-b793-111e316e9cc1\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " Nov 03 23:10:24.229760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.229384 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-xks-ca\") pod \"b493fce4-3699-4660-b793-111e316e9cc1\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " Nov 03 23:10:24.229760 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.229569 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b493fce4-3699-4660-b793-111e316e9cc1-kube-api-access-9wsgp\" (UniqueName: \"kubernetes.io/projected/b493fce4-3699-4660-b793-111e316e9cc1-kube-api-access-9wsgp\") pod \"b493fce4-3699-4660-b793-111e316e9cc1\" (UID: \"b493fce4-3699-4660-b793-111e316e9cc1\") " Nov 03 23:10:24.230773 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.230156 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-openshift-ca" pod "b493fce4-3699-4660-b793-111e316e9cc1" (UID: "b493fce4-3699-4660-b793-111e316e9cc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:24.230773 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.230705 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:24.232537 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.232519 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-xks-ca" pod "b493fce4-3699-4660-b793-111e316e9cc1" (UID: "b493fce4-3699-4660-b793-111e316e9cc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:24.285751 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.267208 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b493fce4-3699-4660-b793-111e316e9cc1-kube-api-access-9wsgp" pod "b493fce4-3699-4660-b793-111e316e9cc1" (UID: "b493fce4-3699-4660-b793-111e316e9cc1"). InnerVolumeSpecName "kube-api-access-9wsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:24.332020 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.331947 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wsgp\" (UniqueName: \"kubernetes.io/projected/b493fce4-3699-4660-b793-111e316e9cc1-kube-api-access-9wsgp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:24.332020 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.331986 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493fce4-3699-4660-b793-111e316e9cc1-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:24.484495 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.484380 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" event={"ID":"b493fce4-3699-4660-b793-111e316e9cc1","Type":"ContainerDied","Data":"8b443141eedf27b219abdf1b245dbc471794e7bdf82e450411c38d671523bae0"} Nov 03 23:10:24.484819 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.484777 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b443141eedf27b219abdf1b245dbc471794e7bdf82e450411c38d671523bae0" Nov 03 23:10:24.485110 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:24.485035 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp" Nov 03 23:10:32.187481 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:10:32.187439 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:10:42.480946 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.480864 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm"] Nov 03 23:10:42.481456 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.481334 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b493fce4-3699-4660-b793-111e316e9cc1" containerName="revoke-keys" Nov 03 23:10:42.481456 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.481345 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b493fce4-3699-4660-b793-111e316e9cc1" containerName="revoke-keys" Nov 03 23:10:42.481456 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.481434 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b493fce4-3699-4660-b793-111e316e9cc1" containerName="revoke-keys" Nov 03 23:10:42.484678 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.484651 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.500683 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.500642 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:42.505199 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.505170 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm"] Nov 03 23:10:42.511865 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.511837 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:10:42.532107 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.532080 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.532257 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.532113 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.532257 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.532191 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvwn\" (UniqueName: \"kubernetes.io/projected/205c43a6-7f74-45d7-8f38-27f05578b5ba-kube-api-access-rtvwn\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.633038 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.632994 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.633249 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.633046 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.633249 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.633083 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvwn\" (UniqueName: \"kubernetes.io/projected/205c43a6-7f74-45d7-8f38-27f05578b5ba-kube-api-access-rtvwn\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.633673 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.633637 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.635373 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.635355 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.655349 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.655327 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvwn\" (UniqueName: \"kubernetes.io/projected/205c43a6-7f74-45d7-8f38-27f05578b5ba-kube-api-access-rtvwn\") pod \"maas-api-revoke-sub-ae594979-ck7tm\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:42.797663 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:42.797571 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:43.100662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:43.100628 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm"] Nov 03 23:10:43.104326 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:10:43.104296 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205c43a6_7f74_45d7_8f38_27f05578b5ba.slice/crio-8c30029ecab67e318be4c641b4e249c992f90e7694ac8d32d002bebb5f24e668 WatchSource:0}: Error finding container 8c30029ecab67e318be4c641b4e249c992f90e7694ac8d32d002bebb5f24e668: Status 404 returned error can't find the container with id 8c30029ecab67e318be4c641b4e249c992f90e7694ac8d32d002bebb5f24e668 Nov 03 23:10:43.638829 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:43.638793 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" event={"ID":"205c43a6-7f74-45d7-8f38-27f05578b5ba","Type":"ContainerStarted","Data":"2fb49c48be366f65d266cb28c49ce01b0831427f4939decc172762b779b574e8"} Nov 03 23:10:43.639199 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:43.638844 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" event={"ID":"205c43a6-7f74-45d7-8f38-27f05578b5ba","Type":"ContainerStarted","Data":"8c30029ecab67e318be4c641b4e249c992f90e7694ac8d32d002bebb5f24e668"} Nov 03 23:10:43.677619 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:43.677553 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" podStartSLOduration=1.677536286 podStartE2EDuration="1.677536286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:10:43.675867899 +0000 UTC m=+3465.105203891" watchObservedRunningTime="2026-11-03 23:10:43.677536286 +0000 UTC m=+3465.106872280" Nov 03 23:10:44.646314 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:44.646281 4625 generic.go:356] "Generic (PLEG): container finished" podID="205c43a6-7f74-45d7-8f38-27f05578b5ba" containerID="2fb49c48be366f65d266cb28c49ce01b0831427f4939decc172762b779b574e8" exitCode=0 Nov 03 23:10:44.646745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:44.646354 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" event={"ID":"205c43a6-7f74-45d7-8f38-27f05578b5ba","Type":"ContainerDied","Data":"2fb49c48be366f65d266cb28c49ce01b0831427f4939decc172762b779b574e8"} Nov 03 23:10:45.780761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.780736 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:45.862157 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.862128 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205c43a6-7f74-45d7-8f38-27f05578b5ba-kube-api-access-rtvwn\" (UniqueName: \"kubernetes.io/projected/205c43a6-7f74-45d7-8f38-27f05578b5ba-kube-api-access-rtvwn\") pod \"205c43a6-7f74-45d7-8f38-27f05578b5ba\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " Nov 03 23:10:45.862331 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.862192 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-openshift-ca\") pod \"205c43a6-7f74-45d7-8f38-27f05578b5ba\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " Nov 03 23:10:45.862331 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.862247 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-xks-ca\") pod \"205c43a6-7f74-45d7-8f38-27f05578b5ba\" (UID: \"205c43a6-7f74-45d7-8f38-27f05578b5ba\") " Nov 03 23:10:45.862561 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.862540 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-openshift-ca" pod "205c43a6-7f74-45d7-8f38-27f05578b5ba" (UID: "205c43a6-7f74-45d7-8f38-27f05578b5ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:45.862650 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.862633 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:45.864272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.864250 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205c43a6-7f74-45d7-8f38-27f05578b5ba-kube-api-access-rtvwn" pod "205c43a6-7f74-45d7-8f38-27f05578b5ba" (UID: "205c43a6-7f74-45d7-8f38-27f05578b5ba"). InnerVolumeSpecName "kube-api-access-rtvwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:45.864272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.864263 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-xks-ca" pod "205c43a6-7f74-45d7-8f38-27f05578b5ba" (UID: "205c43a6-7f74-45d7-8f38-27f05578b5ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:45.963199 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.963116 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvwn\" (UniqueName: \"kubernetes.io/projected/205c43a6-7f74-45d7-8f38-27f05578b5ba-kube-api-access-rtvwn\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:45.963199 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:45.963146 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205c43a6-7f74-45d7-8f38-27f05578b5ba-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:10:46.659379 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:46.659343 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" Nov 03 23:10:46.659585 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:46.659341 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm" event={"ID":"205c43a6-7f74-45d7-8f38-27f05578b5ba","Type":"ContainerDied","Data":"8c30029ecab67e318be4c641b4e249c992f90e7694ac8d32d002bebb5f24e668"} Nov 03 23:10:46.659585 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:10:46.659464 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c30029ecab67e318be4c641b4e249c992f90e7694ac8d32d002bebb5f24e668" Nov 03 23:11:02.188773 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:11:02.185872 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:11:04.798390 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.798357 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz"] Nov 03 23:11:04.799082 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.799060 4625 cpu_manager.go:414] "containerMap: removing container" podUID="205c43a6-7f74-45d7-8f38-27f05578b5ba" containerName="revoke-keys" Nov 03 23:11:04.799193 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.799084 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205c43a6-7f74-45d7-8f38-27f05578b5ba" containerName="revoke-keys" Nov 03 23:11:04.799253 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.799222 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="205c43a6-7f74-45d7-8f38-27f05578b5ba" containerName="revoke-keys" Nov 03 23:11:04.802384 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.802364 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.821588 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.821557 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz"] Nov 03 23:11:04.828926 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.828889 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:11:04.829051 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.828937 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:11:04.873323 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.873292 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxsm5\" (UniqueName: \"kubernetes.io/projected/539fdf04-bf7e-47cc-8374-6d60692dd540-kube-api-access-pxsm5\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.873501 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.873457 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.873551 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.873529 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.974406 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.974366 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxsm5\" (UniqueName: \"kubernetes.io/projected/539fdf04-bf7e-47cc-8374-6d60692dd540-kube-api-access-pxsm5\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.974616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.974463 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.974616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.974503 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.975249 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.975222 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.976875 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.976848 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:04.996930 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:04.996905 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxsm5\" (UniqueName: \"kubernetes.io/projected/539fdf04-bf7e-47cc-8374-6d60692dd540-kube-api-access-pxsm5\") pod \"maas-api-revoke-sub-a4aef7d7-dczvz\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:05.115749 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:05.115698 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:05.418517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:05.418345 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz"] Nov 03 23:11:05.421348 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:11:05.421318 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539fdf04_bf7e_47cc_8374_6d60692dd540.slice/crio-73b575721ba522aa6cdf582f9e1b3f01a0e00e538579188f06da2447a7d21ad4 WatchSource:0}: Error finding container 73b575721ba522aa6cdf582f9e1b3f01a0e00e538579188f06da2447a7d21ad4: Status 404 returned error can't find the container with id 73b575721ba522aa6cdf582f9e1b3f01a0e00e538579188f06da2447a7d21ad4 Nov 03 23:11:05.775627 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:05.775596 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" event={"ID":"539fdf04-bf7e-47cc-8374-6d60692dd540","Type":"ContainerStarted","Data":"1b97e8cdcbb60d005bbe3fb81420a56596262b25c262e081d7e104080dce1fd1"} Nov 03 23:11:05.775811 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:05.775654 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" event={"ID":"539fdf04-bf7e-47cc-8374-6d60692dd540","Type":"ContainerStarted","Data":"73b575721ba522aa6cdf582f9e1b3f01a0e00e538579188f06da2447a7d21ad4"} Nov 03 23:11:05.814423 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:05.814373 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" podStartSLOduration=1.814359605 podStartE2EDuration="1.814359605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:05.813484815 +0000 UTC m=+3487.242820806" watchObservedRunningTime="2026-11-03 23:11:05.814359605 +0000 UTC m=+3487.243695588" Nov 03 23:11:06.782692 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:06.782661 4625 generic.go:356] "Generic (PLEG): container finished" podID="539fdf04-bf7e-47cc-8374-6d60692dd540" containerID="1b97e8cdcbb60d005bbe3fb81420a56596262b25c262e081d7e104080dce1fd1" exitCode=0 Nov 03 23:11:06.782907 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:06.782753 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" event={"ID":"539fdf04-bf7e-47cc-8374-6d60692dd540","Type":"ContainerDied","Data":"1b97e8cdcbb60d005bbe3fb81420a56596262b25c262e081d7e104080dce1fd1"} Nov 03 23:11:07.930795 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:07.925584 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:08.008811 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.008782 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-openshift-ca\") pod \"539fdf04-bf7e-47cc-8374-6d60692dd540\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " Nov 03 23:11:08.008976 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.008908 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-xks-ca\") pod \"539fdf04-bf7e-47cc-8374-6d60692dd540\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " Nov 03 23:11:08.008976 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.008936 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539fdf04-bf7e-47cc-8374-6d60692dd540-kube-api-access-pxsm5\" (UniqueName: \"kubernetes.io/projected/539fdf04-bf7e-47cc-8374-6d60692dd540-kube-api-access-pxsm5\") pod \"539fdf04-bf7e-47cc-8374-6d60692dd540\" (UID: \"539fdf04-bf7e-47cc-8374-6d60692dd540\") " Nov 03 23:11:08.009157 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.009134 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-openshift-ca" pod "539fdf04-bf7e-47cc-8374-6d60692dd540" (UID: "539fdf04-bf7e-47cc-8374-6d60692dd540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:08.011071 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.011043 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-xks-ca" pod "539fdf04-bf7e-47cc-8374-6d60692dd540" (UID: "539fdf04-bf7e-47cc-8374-6d60692dd540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:08.011071 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.011054 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539fdf04-bf7e-47cc-8374-6d60692dd540-kube-api-access-pxsm5" pod "539fdf04-bf7e-47cc-8374-6d60692dd540" (UID: "539fdf04-bf7e-47cc-8374-6d60692dd540"). InnerVolumeSpecName "kube-api-access-pxsm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:08.110517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.110487 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:08.110517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.110516 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539fdf04-bf7e-47cc-8374-6d60692dd540-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:08.110696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.110525 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxsm5\" (UniqueName: \"kubernetes.io/projected/539fdf04-bf7e-47cc-8374-6d60692dd540-kube-api-access-pxsm5\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:08.795490 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.795455 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" event={"ID":"539fdf04-bf7e-47cc-8374-6d60692dd540","Type":"ContainerDied","Data":"73b575721ba522aa6cdf582f9e1b3f01a0e00e538579188f06da2447a7d21ad4"} Nov 03 23:11:08.795657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.795503 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz" Nov 03 23:11:08.795657 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:08.795510 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b575721ba522aa6cdf582f9e1b3f01a0e00e538579188f06da2447a7d21ad4" Nov 03 23:11:26.054753 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.054707 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2"] Nov 03 23:11:26.055241 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.055191 4625 cpu_manager.go:414] "containerMap: removing container" podUID="539fdf04-bf7e-47cc-8374-6d60692dd540" containerName="revoke-keys" Nov 03 23:11:26.055241 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.055203 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539fdf04-bf7e-47cc-8374-6d60692dd540" containerName="revoke-keys" Nov 03 23:11:26.055319 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.055310 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="539fdf04-bf7e-47cc-8374-6d60692dd540" containerName="revoke-keys" Nov 03 23:11:26.058885 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.058861 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.086599 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.086562 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:11:26.086778 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.086675 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:11:26.089613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.089582 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2"] Nov 03 23:11:26.113338 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.113300 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqg26\" (UniqueName: \"kubernetes.io/projected/a4483708-7c3b-436c-837a-359bbaa4ffd1-kube-api-access-bqg26\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.113536 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.113446 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.113536 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.113522 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.214632 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.214602 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.214854 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.214645 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.214854 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.214696 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqg26\" (UniqueName: \"kubernetes.io/projected/a4483708-7c3b-436c-837a-359bbaa4ffd1-kube-api-access-bqg26\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.215394 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.215366 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.216996 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.216968 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.236915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.236892 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqg26\" (UniqueName: \"kubernetes.io/projected/a4483708-7c3b-436c-837a-359bbaa4ffd1-kube-api-access-bqg26\") pod \"maas-api-revoke-sub-79805ca9-w76s2\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.371988 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.371951 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:26.673562 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.673384 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2"] Nov 03 23:11:26.676422 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:11:26.676394 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4483708_7c3b_436c_837a_359bbaa4ffd1.slice/crio-526b0dd63f954063c2d71750d32551da31eaa95a5b5a94121c47a37937e95505 WatchSource:0}: Error finding container 526b0dd63f954063c2d71750d32551da31eaa95a5b5a94121c47a37937e95505: Status 404 returned error can't find the container with id 526b0dd63f954063c2d71750d32551da31eaa95a5b5a94121c47a37937e95505 Nov 03 23:11:26.939759 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.939647 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" event={"ID":"a4483708-7c3b-436c-837a-359bbaa4ffd1","Type":"ContainerStarted","Data":"f29f5875d50ef2819ef72c3b5577204dbadb764543f962a16095efce1b6e3f9e"} Nov 03 23:11:26.939759 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.939707 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" event={"ID":"a4483708-7c3b-436c-837a-359bbaa4ffd1","Type":"ContainerStarted","Data":"526b0dd63f954063c2d71750d32551da31eaa95a5b5a94121c47a37937e95505"} Nov 03 23:11:26.978196 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:26.978100 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" podStartSLOduration=0.978082272 podStartE2EDuration="978.082272ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:26.976874664 +0000 UTC m=+3508.406210656" watchObservedRunningTime="2026-11-03 23:11:26.978082272 +0000 UTC m=+3508.407418265" Nov 03 23:11:27.946728 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:27.946683 4625 generic.go:356] "Generic (PLEG): container finished" podID="a4483708-7c3b-436c-837a-359bbaa4ffd1" containerID="f29f5875d50ef2819ef72c3b5577204dbadb764543f962a16095efce1b6e3f9e" exitCode=0 Nov 03 23:11:27.947131 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:27.946777 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" event={"ID":"a4483708-7c3b-436c-837a-359bbaa4ffd1","Type":"ContainerDied","Data":"f29f5875d50ef2819ef72c3b5577204dbadb764543f962a16095efce1b6e3f9e"} Nov 03 23:11:29.091451 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.091426 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:29.142658 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.142623 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-openshift-ca\") pod \"a4483708-7c3b-436c-837a-359bbaa4ffd1\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " Nov 03 23:11:29.142893 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.142742 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4483708-7c3b-436c-837a-359bbaa4ffd1-kube-api-access-bqg26\" (UniqueName: \"kubernetes.io/projected/a4483708-7c3b-436c-837a-359bbaa4ffd1-kube-api-access-bqg26\") pod \"a4483708-7c3b-436c-837a-359bbaa4ffd1\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " Nov 03 23:11:29.142893 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.142786 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-xks-ca\") pod \"a4483708-7c3b-436c-837a-359bbaa4ffd1\" (UID: \"a4483708-7c3b-436c-837a-359bbaa4ffd1\") " Nov 03 23:11:29.143056 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.143030 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-openshift-ca" pod "a4483708-7c3b-436c-837a-359bbaa4ffd1" (UID: "a4483708-7c3b-436c-837a-359bbaa4ffd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:29.143132 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.143117 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:29.144956 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.144926 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-xks-ca" pod "a4483708-7c3b-436c-837a-359bbaa4ffd1" (UID: "a4483708-7c3b-436c-837a-359bbaa4ffd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:29.145245 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.145225 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4483708-7c3b-436c-837a-359bbaa4ffd1-kube-api-access-bqg26" pod "a4483708-7c3b-436c-837a-359bbaa4ffd1" (UID: "a4483708-7c3b-436c-837a-359bbaa4ffd1"). InnerVolumeSpecName "kube-api-access-bqg26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:29.243813 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.243743 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4483708-7c3b-436c-837a-359bbaa4ffd1-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:29.243813 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.243771 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqg26\" (UniqueName: \"kubernetes.io/projected/a4483708-7c3b-436c-837a-359bbaa4ffd1-kube-api-access-bqg26\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:29.959982 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.959953 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" Nov 03 23:11:29.960169 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.959989 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2" event={"ID":"a4483708-7c3b-436c-837a-359bbaa4ffd1","Type":"ContainerDied","Data":"526b0dd63f954063c2d71750d32551da31eaa95a5b5a94121c47a37937e95505"} Nov 03 23:11:29.960169 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:29.960039 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="526b0dd63f954063c2d71750d32551da31eaa95a5b5a94121c47a37937e95505" Nov 03 23:11:32.184707 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:11:32.184671 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:11:42.978641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:42.978599 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx"] Nov 03 23:11:42.979167 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:42.979045 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a4483708-7c3b-436c-837a-359bbaa4ffd1" containerName="revoke-keys" Nov 03 23:11:42.979167 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:42.979057 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4483708-7c3b-436c-837a-359bbaa4ffd1" containerName="revoke-keys" Nov 03 23:11:42.979167 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:42.979135 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4483708-7c3b-436c-837a-359bbaa4ffd1" containerName="revoke-keys" Nov 03 23:11:42.982484 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:42.982460 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:42.997971 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:42.997942 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:11:43.009259 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.009230 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:11:43.012348 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.012322 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx"] Nov 03 23:11:43.112570 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.112533 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.112796 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.112700 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.112898 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.112810 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rr8m\" (UniqueName: \"kubernetes.io/projected/674c0ae7-5f93-4c7b-9b60-43bb0643e018-kube-api-access-7rr8m\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.214020 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.213985 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.214212 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.214037 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rr8m\" (UniqueName: \"kubernetes.io/projected/674c0ae7-5f93-4c7b-9b60-43bb0643e018-kube-api-access-7rr8m\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.214212 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.214154 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.214691 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.214667 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.216383 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.216360 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.236351 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.236295 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rr8m\" (UniqueName: \"kubernetes.io/projected/674c0ae7-5f93-4c7b-9b60-43bb0643e018-kube-api-access-7rr8m\") pod \"maas-api-revoke-sub-0b7e8edb-fzkrx\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.295362 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.295331 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:43.599483 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:43.599459 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx"] Nov 03 23:11:43.601683 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:11:43.601658 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674c0ae7_5f93_4c7b_9b60_43bb0643e018.slice/crio-da7144a0faea82332f5ed2343dbc7573d14dc959a6b70ba363dbf70fa7b02a20 WatchSource:0}: Error finding container da7144a0faea82332f5ed2343dbc7573d14dc959a6b70ba363dbf70fa7b02a20: Status 404 returned error can't find the container with id da7144a0faea82332f5ed2343dbc7573d14dc959a6b70ba363dbf70fa7b02a20 Nov 03 23:11:44.048792 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:44.048755 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" event={"ID":"674c0ae7-5f93-4c7b-9b60-43bb0643e018","Type":"ContainerStarted","Data":"db9c12f2f9efe1f44740def47c38b0c7ca5243ef43f3cfe44116ff22c66ec823"} Nov 03 23:11:44.049250 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:44.048811 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" event={"ID":"674c0ae7-5f93-4c7b-9b60-43bb0643e018","Type":"ContainerStarted","Data":"da7144a0faea82332f5ed2343dbc7573d14dc959a6b70ba363dbf70fa7b02a20"} Nov 03 23:11:44.088167 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:44.088110 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" podStartSLOduration=2.088096118 podStartE2EDuration="2.088096118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:44.08567127 +0000 UTC m=+3525.515007261" watchObservedRunningTime="2026-11-03 23:11:44.088096118 +0000 UTC m=+3525.517432148" Nov 03 23:11:45.056529 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:45.056500 4625 generic.go:356] "Generic (PLEG): container finished" podID="674c0ae7-5f93-4c7b-9b60-43bb0643e018" containerID="db9c12f2f9efe1f44740def47c38b0c7ca5243ef43f3cfe44116ff22c66ec823" exitCode=0 Nov 03 23:11:45.056936 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:45.056585 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" event={"ID":"674c0ae7-5f93-4c7b-9b60-43bb0643e018","Type":"ContainerDied","Data":"db9c12f2f9efe1f44740def47c38b0c7ca5243ef43f3cfe44116ff22c66ec823"} Nov 03 23:11:46.188524 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.188495 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:46.346805 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.346770 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-openshift-ca\") pod \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " Nov 03 23:11:46.347004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.346867 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674c0ae7-5f93-4c7b-9b60-43bb0643e018-kube-api-access-7rr8m\" (UniqueName: \"kubernetes.io/projected/674c0ae7-5f93-4c7b-9b60-43bb0643e018-kube-api-access-7rr8m\") pod \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " Nov 03 23:11:46.347004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.346915 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-xks-ca\") pod \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\" (UID: \"674c0ae7-5f93-4c7b-9b60-43bb0643e018\") " Nov 03 23:11:46.347142 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.347119 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-openshift-ca" pod "674c0ae7-5f93-4c7b-9b60-43bb0643e018" (UID: "674c0ae7-5f93-4c7b-9b60-43bb0643e018"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:46.347298 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.347282 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:46.349025 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.348997 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-xks-ca" pod "674c0ae7-5f93-4c7b-9b60-43bb0643e018" (UID: "674c0ae7-5f93-4c7b-9b60-43bb0643e018"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:46.349141 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.349028 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674c0ae7-5f93-4c7b-9b60-43bb0643e018-kube-api-access-7rr8m" pod "674c0ae7-5f93-4c7b-9b60-43bb0643e018" (UID: "674c0ae7-5f93-4c7b-9b60-43bb0643e018"). InnerVolumeSpecName "kube-api-access-7rr8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:46.448745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.448688 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rr8m\" (UniqueName: \"kubernetes.io/projected/674c0ae7-5f93-4c7b-9b60-43bb0643e018-kube-api-access-7rr8m\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:46.448745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:46.448744 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674c0ae7-5f93-4c7b-9b60-43bb0643e018-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:47.068696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:47.068610 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" event={"ID":"674c0ae7-5f93-4c7b-9b60-43bb0643e018","Type":"ContainerDied","Data":"da7144a0faea82332f5ed2343dbc7573d14dc959a6b70ba363dbf70fa7b02a20"} Nov 03 23:11:47.068696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:47.068630 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx" Nov 03 23:11:47.068696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:47.068660 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da7144a0faea82332f5ed2343dbc7573d14dc959a6b70ba363dbf70fa7b02a20" Nov 03 23:11:53.829027 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.828990 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h"] Nov 03 23:11:53.829534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.829418 4625 cpu_manager.go:414] "containerMap: removing container" podUID="674c0ae7-5f93-4c7b-9b60-43bb0643e018" containerName="revoke-keys" Nov 03 23:11:53.829534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.829429 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674c0ae7-5f93-4c7b-9b60-43bb0643e018" containerName="revoke-keys" Nov 03 23:11:53.829534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.829524 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="674c0ae7-5f93-4c7b-9b60-43bb0643e018" containerName="revoke-keys" Nov 03 23:11:53.833815 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.833792 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:53.848577 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.848535 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:11:53.854036 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.853997 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h"] Nov 03 23:11:53.859825 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.859790 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:11:53.921390 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.921359 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:53.921561 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.921404 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflrj\" (UniqueName: \"kubernetes.io/projected/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-kube-api-access-bflrj\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:53.921561 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:53.921420 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.022346 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.022306 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflrj\" (UniqueName: \"kubernetes.io/projected/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-kube-api-access-bflrj\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.022599 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.022570 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.022802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.022784 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.023293 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.023261 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.025078 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.025059 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.044959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.044930 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflrj\" (UniqueName: \"kubernetes.io/projected/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-kube-api-access-bflrj\") pod \"maas-api-revoke-sub-3d4156f1-fdh2h\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.147136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.147106 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:54.452504 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:54.452475 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h"] Nov 03 23:11:54.455131 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:11:54.455092 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c5f676_8fd5_40ea_a604_1b5af9cd6e0a.slice/crio-5618eb9b9e5d9d4750608e3c7f9fd2c6e908f5367d93c0d33c9d563febc5010b WatchSource:0}: Error finding container 5618eb9b9e5d9d4750608e3c7f9fd2c6e908f5367d93c0d33c9d563febc5010b: Status 404 returned error can't find the container with id 5618eb9b9e5d9d4750608e3c7f9fd2c6e908f5367d93c0d33c9d563febc5010b Nov 03 23:11:55.116469 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:55.116436 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" event={"ID":"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a","Type":"ContainerStarted","Data":"8bb51dd80a10098678e6a07b74f99abecfaa94a354ceed19c9c3a185bfe6ff83"} Nov 03 23:11:55.116905 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:55.116487 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" event={"ID":"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a","Type":"ContainerStarted","Data":"5618eb9b9e5d9d4750608e3c7f9fd2c6e908f5367d93c0d33c9d563febc5010b"} Nov 03 23:11:55.156642 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:55.156592 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" podStartSLOduration=2.156578391 podStartE2EDuration="2.156578391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:55.153384401 +0000 UTC m=+3536.582720392" watchObservedRunningTime="2026-11-03 23:11:55.156578391 +0000 UTC m=+3536.585914382" Nov 03 23:11:56.123700 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:56.123661 4625 generic.go:356] "Generic (PLEG): container finished" podID="f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" containerID="8bb51dd80a10098678e6a07b74f99abecfaa94a354ceed19c9c3a185bfe6ff83" exitCode=0 Nov 03 23:11:56.124120 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:56.123741 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" event={"ID":"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a","Type":"ContainerDied","Data":"8bb51dd80a10098678e6a07b74f99abecfaa94a354ceed19c9c3a185bfe6ff83"} Nov 03 23:11:57.265031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.265002 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:57.352428 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.352396 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-xks-ca\") pod \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " Nov 03 23:11:57.352616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.352520 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-openshift-ca\") pod \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " Nov 03 23:11:57.352616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.352561 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-kube-api-access-bflrj\" (UniqueName: \"kubernetes.io/projected/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-kube-api-access-bflrj\") pod \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\" (UID: \"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a\") " Nov 03 23:11:57.352933 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.352907 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-openshift-ca" pod "f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" (UID: "f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:57.354580 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.354548 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-kube-api-access-bflrj" pod "f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" (UID: "f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a"). InnerVolumeSpecName "kube-api-access-bflrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:57.354580 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.354570 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-xks-ca" pod "f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" (UID: "f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:57.453770 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.453662 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:57.453770 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.453692 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflrj\" (UniqueName: \"kubernetes.io/projected/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-kube-api-access-bflrj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:57.453770 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:57.453706 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:11:58.137542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:58.137509 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" Nov 03 23:11:58.137744 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:58.137537 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h" event={"ID":"f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a","Type":"ContainerDied","Data":"5618eb9b9e5d9d4750608e3c7f9fd2c6e908f5367d93c0d33c9d563febc5010b"} Nov 03 23:11:58.137744 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:11:58.137586 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5618eb9b9e5d9d4750608e3c7f9fd2c6e908f5367d93c0d33c9d563febc5010b" Nov 03 23:12:02.183529 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:12:02.183491 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:12:08.201031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:08.200980 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zqt58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:12:08.201753 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:08.201679 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqt58" podUID="ef90202f-42a4-45ca-93b0-deebd7306fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:12:12.029370 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:12.029281 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9"] Nov 03 23:12:12.041652 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:12.041622 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789f5be1-7nrb9"] Nov 03 23:12:12.727838 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:12.727803 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee74ee7-e9fa-4677-9f25-174440a86b5d" path="/var/lib/kubelet/pods/6ee74ee7-e9fa-4677-9f25-174440a86b5d/volumes" Nov 03 23:12:14.114792 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.114755 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h"] Nov 03 23:12:14.125633 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.125604 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdh2h"] Nov 03 23:12:14.164066 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.164034 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q"] Nov 03 23:12:14.164484 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.164470 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" containerName="revoke-keys" Nov 03 23:12:14.164484 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.164484 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" containerName="revoke-keys" Nov 03 23:12:14.164667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.164562 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" containerName="revoke-keys" Nov 03 23:12:14.168007 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.167992 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.183459 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.183433 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:12:14.183664 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.183459 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:12:14.187072 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.187053 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q"] Nov 03 23:12:14.262968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.262937 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.262968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.262971 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.263188 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.263000 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25sr7\" (UniqueName: \"kubernetes.io/projected/b4c1ef94-d841-44cc-add1-da7e79935848-kube-api-access-25sr7\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.363959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.363930 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.363959 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.363962 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.364158 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.363995 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25sr7\" (UniqueName: \"kubernetes.io/projected/b4c1ef94-d841-44cc-add1-da7e79935848-kube-api-access-25sr7\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.364685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.364666 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.366245 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.366205 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.386440 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.386408 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25sr7\" (UniqueName: \"kubernetes.io/projected/b4c1ef94-d841-44cc-add1-da7e79935848-kube-api-access-25sr7\") pod \"maas-api-revoke-sub-3d4156f1-mww9q\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.480218 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.480182 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:14.727257 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.727186 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a" path="/var/lib/kubelet/pods/f5c5f676-8fd5-40ea-a604-1b5af9cd6e0a/volumes" Nov 03 23:12:14.783210 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:14.783178 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q"] Nov 03 23:12:14.793391 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:12:14.793359 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c1ef94_d841_44cc_add1_da7e79935848.slice/crio-2d6e7fb45738397d28b15d67d3983f70dc19e75704a9dbef6c416136dc01e180 WatchSource:0}: Error finding container 2d6e7fb45738397d28b15d67d3983f70dc19e75704a9dbef6c416136dc01e180: Status 404 returned error can't find the container with id 2d6e7fb45738397d28b15d67d3983f70dc19e75704a9dbef6c416136dc01e180 Nov 03 23:12:15.355235 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:15.355193 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" event={"ID":"b4c1ef94-d841-44cc-add1-da7e79935848","Type":"ContainerStarted","Data":"9a29602dee6f781880769125ab86fc02bc802cd2c0a951543f763598667a9ac3"} Nov 03 23:12:15.355661 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:15.355268 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" event={"ID":"b4c1ef94-d841-44cc-add1-da7e79935848","Type":"ContainerStarted","Data":"2d6e7fb45738397d28b15d67d3983f70dc19e75704a9dbef6c416136dc01e180"} Nov 03 23:12:15.393922 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:15.393865 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" podStartSLOduration=1.393847008 podStartE2EDuration="1.393847008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:15.392913805 +0000 UTC m=+3556.822249787" watchObservedRunningTime="2026-11-03 23:12:15.393847008 +0000 UTC m=+3556.823183001" Nov 03 23:12:16.363206 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:16.363170 4625 generic.go:356] "Generic (PLEG): container finished" podID="b4c1ef94-d841-44cc-add1-da7e79935848" containerID="9a29602dee6f781880769125ab86fc02bc802cd2c0a951543f763598667a9ac3" exitCode=0 Nov 03 23:12:16.363596 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:16.363263 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" event={"ID":"b4c1ef94-d841-44cc-add1-da7e79935848","Type":"ContainerDied","Data":"9a29602dee6f781880769125ab86fc02bc802cd2c0a951543f763598667a9ac3"} Nov 03 23:12:17.520822 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.520796 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:17.591676 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.591644 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-xks-ca\") pod \"b4c1ef94-d841-44cc-add1-da7e79935848\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " Nov 03 23:12:17.591676 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.591682 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-openshift-ca\") pod \"b4c1ef94-d841-44cc-add1-da7e79935848\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " Nov 03 23:12:17.591927 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.591710 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c1ef94-d841-44cc-add1-da7e79935848-kube-api-access-25sr7\" (UniqueName: \"kubernetes.io/projected/b4c1ef94-d841-44cc-add1-da7e79935848-kube-api-access-25sr7\") pod \"b4c1ef94-d841-44cc-add1-da7e79935848\" (UID: \"b4c1ef94-d841-44cc-add1-da7e79935848\") " Nov 03 23:12:17.592059 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.592033 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-openshift-ca" pod "b4c1ef94-d841-44cc-add1-da7e79935848" (UID: "b4c1ef94-d841-44cc-add1-da7e79935848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:12:17.593759 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.593733 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-xks-ca" pod "b4c1ef94-d841-44cc-add1-da7e79935848" (UID: "b4c1ef94-d841-44cc-add1-da7e79935848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:17.593897 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.593878 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c1ef94-d841-44cc-add1-da7e79935848-kube-api-access-25sr7" pod "b4c1ef94-d841-44cc-add1-da7e79935848" (UID: "b4c1ef94-d841-44cc-add1-da7e79935848"). InnerVolumeSpecName "kube-api-access-25sr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:12:17.693028 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.692949 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:12:17.693028 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.692975 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1ef94-d841-44cc-add1-da7e79935848-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:12:17.693028 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:17.692985 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25sr7\" (UniqueName: \"kubernetes.io/projected/b4c1ef94-d841-44cc-add1-da7e79935848-kube-api-access-25sr7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:12:18.376114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:18.376080 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" Nov 03 23:12:18.376271 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:18.376111 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q" event={"ID":"b4c1ef94-d841-44cc-add1-da7e79935848","Type":"ContainerDied","Data":"2d6e7fb45738397d28b15d67d3983f70dc19e75704a9dbef6c416136dc01e180"} Nov 03 23:12:18.376271 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:18.376159 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d6e7fb45738397d28b15d67d3983f70dc19e75704a9dbef6c416136dc01e180" Nov 03 23:12:28.033926 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:28.033888 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z"] Nov 03 23:12:28.044985 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:28.044945 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e22966d-s9z5z"] Nov 03 23:12:28.727385 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:28.727357 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e36eab-2abd-4a73-b86d-be125deaa935" path="/var/lib/kubelet/pods/00e36eab-2abd-4a73-b86d-be125deaa935/volumes" Nov 03 23:12:32.186451 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:12:32.186418 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:12:47.889367 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.889329 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt"] Nov 03 23:12:47.889788 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.889770 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b4c1ef94-d841-44cc-add1-da7e79935848" containerName="revoke-keys" Nov 03 23:12:47.889845 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.889791 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c1ef94-d841-44cc-add1-da7e79935848" containerName="revoke-keys" Nov 03 23:12:47.889910 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.889901 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c1ef94-d841-44cc-add1-da7e79935848" containerName="revoke-keys" Nov 03 23:12:47.893150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.893133 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:47.908691 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.908665 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:12:47.912776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.912750 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt"] Nov 03 23:12:47.920016 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.919958 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:12:47.950624 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.950599 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vb7h\" (UniqueName: \"kubernetes.io/projected/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-kube-api-access-4vb7h\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:47.950794 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.950642 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:47.950794 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:47.950772 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.051525 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.051486 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.051743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.051633 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vb7h\" (UniqueName: \"kubernetes.io/projected/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-kube-api-access-4vb7h\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.051743 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.051682 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.052160 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.052140 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.053901 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.053881 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.073454 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.073433 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vb7h\" (UniqueName: \"kubernetes.io/projected/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-kube-api-access-4vb7h\") pod \"maas-api-revoke-sub-e2fc9c5b-tqlqt\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.206044 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.205966 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:48.508529 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.508500 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt"] Nov 03 23:12:48.511011 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:12:48.510981 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a40fe5_b941_466f_a0e9_fdad9af3eb9b.slice/crio-dbbca8969d4957e360e4a3522f596de37cdcbd5913cc80ba5cc9027155a13e9a WatchSource:0}: Error finding container dbbca8969d4957e360e4a3522f596de37cdcbd5913cc80ba5cc9027155a13e9a: Status 404 returned error can't find the container with id dbbca8969d4957e360e4a3522f596de37cdcbd5913cc80ba5cc9027155a13e9a Nov 03 23:12:48.565795 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:48.565754 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" event={"ID":"46a40fe5-b941-466f-a0e9-fdad9af3eb9b","Type":"ContainerStarted","Data":"dbbca8969d4957e360e4a3522f596de37cdcbd5913cc80ba5cc9027155a13e9a"} Nov 03 23:12:49.573899 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:49.573861 4625 generic.go:356] "Generic (PLEG): container finished" podID="46a40fe5-b941-466f-a0e9-fdad9af3eb9b" containerID="49c1161694e063fd5d6d603c5cd148dc33986766c64841a6e3f29e152f97ae77" exitCode=0 Nov 03 23:12:49.574378 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:49.573942 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" event={"ID":"46a40fe5-b941-466f-a0e9-fdad9af3eb9b","Type":"ContainerDied","Data":"49c1161694e063fd5d6d603c5cd148dc33986766c64841a6e3f29e152f97ae77"} Nov 03 23:12:50.731766 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.731739 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:50.779093 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.779060 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-kube-api-access-4vb7h\" (UniqueName: \"kubernetes.io/projected/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-kube-api-access-4vb7h\") pod \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " Nov 03 23:12:50.779279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.779110 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-xks-ca\") pod \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " Nov 03 23:12:50.779279 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.779224 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-openshift-ca\") pod \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\" (UID: \"46a40fe5-b941-466f-a0e9-fdad9af3eb9b\") " Nov 03 23:12:50.780512 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.780104 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-openshift-ca" pod "46a40fe5-b941-466f-a0e9-fdad9af3eb9b" (UID: "46a40fe5-b941-466f-a0e9-fdad9af3eb9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:12:50.781932 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.781870 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-kube-api-access-4vb7h" pod "46a40fe5-b941-466f-a0e9-fdad9af3eb9b" (UID: "46a40fe5-b941-466f-a0e9-fdad9af3eb9b"). InnerVolumeSpecName "kube-api-access-4vb7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:12:50.782077 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.781950 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-xks-ca" pod "46a40fe5-b941-466f-a0e9-fdad9af3eb9b" (UID: "46a40fe5-b941-466f-a0e9-fdad9af3eb9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:50.880858 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.880827 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vb7h\" (UniqueName: \"kubernetes.io/projected/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-kube-api-access-4vb7h\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:12:50.880858 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.880855 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:12:50.881042 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:50.880864 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a40fe5-b941-466f-a0e9-fdad9af3eb9b-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:12:51.588096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:51.588057 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" event={"ID":"46a40fe5-b941-466f-a0e9-fdad9af3eb9b","Type":"ContainerDied","Data":"dbbca8969d4957e360e4a3522f596de37cdcbd5913cc80ba5cc9027155a13e9a"} Nov 03 23:12:51.588096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:51.588077 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt" Nov 03 23:12:51.588353 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:51.588105 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbca8969d4957e360e4a3522f596de37cdcbd5913cc80ba5cc9027155a13e9a" Nov 03 23:12:58.227553 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.227523 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk"] Nov 03 23:12:58.228022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.228006 4625 cpu_manager.go:414] "containerMap: removing container" podUID="46a40fe5-b941-466f-a0e9-fdad9af3eb9b" containerName="revoke-keys" Nov 03 23:12:58.228067 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.228023 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a40fe5-b941-466f-a0e9-fdad9af3eb9b" containerName="revoke-keys" Nov 03 23:12:58.228109 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.228095 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a40fe5-b941-466f-a0e9-fdad9af3eb9b" containerName="revoke-keys" Nov 03 23:12:58.231464 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.231445 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.251419 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.251389 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk"] Nov 03 23:12:58.260836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.258096 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:12:58.260836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.258151 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:12:58.347629 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.347594 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvlt\" (UniqueName: \"kubernetes.io/projected/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-kube-api-access-mjvlt\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.347824 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.347656 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.347873 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.347832 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.449086 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.449043 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvlt\" (UniqueName: \"kubernetes.io/projected/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-kube-api-access-mjvlt\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.449258 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.449095 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.449258 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.449156 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.449905 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.449878 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.451638 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.451607 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.471522 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.471498 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvlt\" (UniqueName: \"kubernetes.io/projected/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-kube-api-access-mjvlt\") pod \"maas-api-revoke-sub-5a29efb3-stmlk\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.544224 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.544142 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:12:58.853808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:58.853778 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk"] Nov 03 23:12:58.855347 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:12:58.855314 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe5d8f9_b881_4d54_b40a_5be6c7f7540b.slice/crio-511561c44caee1a6e72d785176872a48c1632d66180ec5b84a238e5e0a5c7cf6 WatchSource:0}: Error finding container 511561c44caee1a6e72d785176872a48c1632d66180ec5b84a238e5e0a5c7cf6: Status 404 returned error can't find the container with id 511561c44caee1a6e72d785176872a48c1632d66180ec5b84a238e5e0a5c7cf6 Nov 03 23:12:59.655883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:59.655776 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" event={"ID":"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b","Type":"ContainerStarted","Data":"720745f3838daff190c5568503bfcc5f2ab3c3911c57dc8e44dce68e7cd99584"} Nov 03 23:12:59.655883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:59.655860 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" event={"ID":"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b","Type":"ContainerStarted","Data":"511561c44caee1a6e72d785176872a48c1632d66180ec5b84a238e5e0a5c7cf6"} Nov 03 23:12:59.697856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:12:59.697796 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" podStartSLOduration=1.6977796550000002 podStartE2EDuration="1.697779655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:59.696088689 +0000 UTC m=+3601.125424682" watchObservedRunningTime="2026-11-03 23:12:59.697779655 +0000 UTC m=+3601.127115647" Nov 03 23:13:00.705953 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:00.705917 4625 generic.go:356] "Generic (PLEG): container finished" podID="0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" containerID="720745f3838daff190c5568503bfcc5f2ab3c3911c57dc8e44dce68e7cd99584" exitCode=0 Nov 03 23:13:00.706762 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:00.706025 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" event={"ID":"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b","Type":"ContainerDied","Data":"720745f3838daff190c5568503bfcc5f2ab3c3911c57dc8e44dce68e7cd99584"} Nov 03 23:13:02.143575 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.143252 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:13:02.200364 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:13:02.200294 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:13:02.232765 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.232735 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-kube-api-access-mjvlt\" (UniqueName: \"kubernetes.io/projected/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-kube-api-access-mjvlt\") pod \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " Nov 03 23:13:02.232968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.232860 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-xks-ca\") pod \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " Nov 03 23:13:02.232968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.232901 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-openshift-ca\") pod \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\" (UID: \"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b\") " Nov 03 23:13:02.233460 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.233403 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-openshift-ca" pod "0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" (UID: "0fe5d8f9-b881-4d54-b40a-5be6c7f7540b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:02.235042 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.235012 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-xks-ca" pod "0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" (UID: "0fe5d8f9-b881-4d54-b40a-5be6c7f7540b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:02.235494 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.235470 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-kube-api-access-mjvlt" pod "0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" (UID: "0fe5d8f9-b881-4d54-b40a-5be6c7f7540b"). InnerVolumeSpecName "kube-api-access-mjvlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:02.334639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.334603 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvlt\" (UniqueName: \"kubernetes.io/projected/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-kube-api-access-mjvlt\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:13:02.334639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.334631 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:13:02.334639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.334640 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:13:02.728685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.728654 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" Nov 03 23:13:02.728907 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.728654 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk" event={"ID":"0fe5d8f9-b881-4d54-b40a-5be6c7f7540b","Type":"ContainerDied","Data":"511561c44caee1a6e72d785176872a48c1632d66180ec5b84a238e5e0a5c7cf6"} Nov 03 23:13:02.728907 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:02.728798 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="511561c44caee1a6e72d785176872a48c1632d66180ec5b84a238e5e0a5c7cf6" Nov 03 23:13:04.006341 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:04.006312 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:13:04.060112 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:04.060085 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:13:15.029157 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:15.029123 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc"] Nov 03 23:13:15.039939 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:15.039916 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q4xhc"] Nov 03 23:13:16.726785 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:16.726753 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16bb7923-7273-4062-a923-92478b8853f8" path="/var/lib/kubelet/pods/16bb7923-7273-4062-a923-92478b8853f8/volumes" Nov 03 23:13:17.455870 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.455837 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q"] Nov 03 23:13:17.466640 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.466604 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mww9q"] Nov 03 23:13:17.502955 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.502925 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5"] Nov 03 23:13:17.503373 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.503361 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" containerName="revoke-keys" Nov 03 23:13:17.503419 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.503375 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" containerName="revoke-keys" Nov 03 23:13:17.503455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.503447 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" containerName="revoke-keys" Nov 03 23:13:17.507892 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.507864 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.523104 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.523080 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:13:17.523314 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.523096 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:13:17.526817 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.526795 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5"] Nov 03 23:13:17.596968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.596940 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.597145 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.596985 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8xx\" (UniqueName: \"kubernetes.io/projected/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-kube-api-access-zr8xx\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.597145 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.597116 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.698172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.698132 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.698374 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.698237 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.698374 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.698267 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8xx\" (UniqueName: \"kubernetes.io/projected/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-kube-api-access-zr8xx\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.698892 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.698866 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.700803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.700779 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.720528 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.720472 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8xx\" (UniqueName: \"kubernetes.io/projected/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-kube-api-access-zr8xx\") pod \"maas-api-revoke-sub-3d4156f1-t7wm5\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:17.821173 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:17.821139 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:18.125542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:18.125515 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5"] Nov 03 23:13:18.128296 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:13:18.128262 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ecc126_1657_4dd7_b5d6_e4d3055842ec.slice/crio-5866d2ef8f6008153658afc99fe2ff1ac2610c601db2262219f8b5b956bc0a30 WatchSource:0}: Error finding container 5866d2ef8f6008153658afc99fe2ff1ac2610c601db2262219f8b5b956bc0a30: Status 404 returned error can't find the container with id 5866d2ef8f6008153658afc99fe2ff1ac2610c601db2262219f8b5b956bc0a30 Nov 03 23:13:18.726883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:18.726811 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c1ef94-d841-44cc-add1-da7e79935848" path="/var/lib/kubelet/pods/b4c1ef94-d841-44cc-add1-da7e79935848/volumes" Nov 03 23:13:18.828231 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:18.828197 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" event={"ID":"b7ecc126-1657-4dd7-b5d6-e4d3055842ec","Type":"ContainerStarted","Data":"f4775c5b2c3dc9781a4826f3fa34f21f4054e118d5a813772e985a53d011030a"} Nov 03 23:13:18.828646 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:18.828255 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" event={"ID":"b7ecc126-1657-4dd7-b5d6-e4d3055842ec","Type":"ContainerStarted","Data":"5866d2ef8f6008153658afc99fe2ff1ac2610c601db2262219f8b5b956bc0a30"} Nov 03 23:13:18.866269 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:18.866212 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" podStartSLOduration=1.866196467 podStartE2EDuration="1.866196467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:13:18.865381835 +0000 UTC m=+3620.294717827" watchObservedRunningTime="2026-11-03 23:13:18.866196467 +0000 UTC m=+3620.295532455" Nov 03 23:13:19.836441 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:19.836405 4625 generic.go:356] "Generic (PLEG): container finished" podID="b7ecc126-1657-4dd7-b5d6-e4d3055842ec" containerID="f4775c5b2c3dc9781a4826f3fa34f21f4054e118d5a813772e985a53d011030a" exitCode=0 Nov 03 23:13:19.836866 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:19.836489 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" event={"ID":"b7ecc126-1657-4dd7-b5d6-e4d3055842ec","Type":"ContainerDied","Data":"f4775c5b2c3dc9781a4826f3fa34f21f4054e118d5a813772e985a53d011030a"} Nov 03 23:13:20.974185 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:20.974157 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:21.134164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.134066 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-xks-ca\") pod \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " Nov 03 23:13:21.134471 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.134450 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-kube-api-access-zr8xx\" (UniqueName: \"kubernetes.io/projected/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-kube-api-access-zr8xx\") pod \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " Nov 03 23:13:21.134661 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.134644 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-openshift-ca\") pod \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\" (UID: \"b7ecc126-1657-4dd7-b5d6-e4d3055842ec\") " Nov 03 23:13:21.135660 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.135635 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-openshift-ca" pod "b7ecc126-1657-4dd7-b5d6-e4d3055842ec" (UID: "b7ecc126-1657-4dd7-b5d6-e4d3055842ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:21.138769 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.138172 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-kube-api-access-zr8xx" pod "b7ecc126-1657-4dd7-b5d6-e4d3055842ec" (UID: "b7ecc126-1657-4dd7-b5d6-e4d3055842ec"). InnerVolumeSpecName "kube-api-access-zr8xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:21.142854 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.142822 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-xks-ca" pod "b7ecc126-1657-4dd7-b5d6-e4d3055842ec" (UID: "b7ecc126-1657-4dd7-b5d6-e4d3055842ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:21.236876 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.236842 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:13:21.236876 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.236873 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:13:21.237085 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.236886 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr8xx\" (UniqueName: \"kubernetes.io/projected/b7ecc126-1657-4dd7-b5d6-e4d3055842ec-kube-api-access-zr8xx\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:13:21.849978 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.849947 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" Nov 03 23:13:21.850136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.849978 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5" event={"ID":"b7ecc126-1657-4dd7-b5d6-e4d3055842ec","Type":"ContainerDied","Data":"5866d2ef8f6008153658afc99fe2ff1ac2610c601db2262219f8b5b956bc0a30"} Nov 03 23:13:21.850136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:21.850026 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5866d2ef8f6008153658afc99fe2ff1ac2610c601db2262219f8b5b956bc0a30" Nov 03 23:13:32.189145 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:13:32.189109 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:13:44.868789 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.868698 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h"] Nov 03 23:13:44.869291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.869172 4625 cpu_manager.go:414] "containerMap: removing container" podUID="b7ecc126-1657-4dd7-b5d6-e4d3055842ec" containerName="revoke-keys" Nov 03 23:13:44.869291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.869184 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ecc126-1657-4dd7-b5d6-e4d3055842ec" containerName="revoke-keys" Nov 03 23:13:44.869291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.869288 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ecc126-1657-4dd7-b5d6-e4d3055842ec" containerName="revoke-keys" Nov 03 23:13:44.872678 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.872655 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.891640 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.891609 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.891855 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.891701 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9sr\" (UniqueName: \"kubernetes.io/projected/c2e6223e-5e32-423e-8dec-841ed7fa45ee-kube-api-access-jh9sr\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.891855 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.891826 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e6223e-5e32-423e-8dec-841ed7fa45ee-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.891984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.891916 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.897254 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.897228 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d5590b-serving-cert\"" Nov 03 23:13:44.906292 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.906263 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h"] Nov 03 23:13:44.906491 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.906473 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qn4p2\"" Nov 03 23:13:44.906589 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.906535 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:13:44.993507 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.993475 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.993704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.993529 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9sr\" (UniqueName: \"kubernetes.io/projected/c2e6223e-5e32-423e-8dec-841ed7fa45ee-kube-api-access-jh9sr\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.993704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.993660 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e6223e-5e32-423e-8dec-841ed7fa45ee-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.993903 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.993776 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.994461 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.994432 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e6223e-5e32-423e-8dec-841ed7fa45ee-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.996171 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.996141 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:44.996287 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:44.996252 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:45.015832 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.015811 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9sr\" (UniqueName: \"kubernetes.io/projected/c2e6223e-5e32-423e-8dec-841ed7fa45ee-kube-api-access-jh9sr\") pod \"maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:45.185977 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.185888 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:45.403399 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.403355 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss"] Nov 03 23:13:45.409246 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.408290 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:45.418683 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.418204 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss"] Nov 03 23:13:45.423242 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.422829 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d5590b\"" Nov 03 23:13:45.424941 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.424644 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4582222a-dockercfg-mfvt8\"" Nov 03 23:13:45.499698 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.499670 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h"] Nov 03 23:13:45.501895 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:13:45.501855 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e6223e_5e32_423e_8dec_841ed7fa45ee.slice/crio-58f46bf300bdacc898fe78eb1e60b22d2ce158c380ed720cc2ff8c1f3b706450 WatchSource:0}: Error finding container 58f46bf300bdacc898fe78eb1e60b22d2ce158c380ed720cc2ff8c1f3b706450: Status 404 returned error can't find the container with id 58f46bf300bdacc898fe78eb1e60b22d2ce158c380ed720cc2ff8c1f3b706450 Nov 03 23:13:45.502012 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.501900 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbdw\" (UniqueName: \"kubernetes.io/projected/0170479a-87ab-464c-accc-a9e8f8fb4a1d-kube-api-access-lxbdw\") pod \"payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:45.502012 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.501976 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0170479a-87ab-464c-accc-a9e8f8fb4a1d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:45.503120 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.503102 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:13:45.602948 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.602917 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbdw\" (UniqueName: \"kubernetes.io/projected/0170479a-87ab-464c-accc-a9e8f8fb4a1d-kube-api-access-lxbdw\") pod \"payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:45.603138 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.602980 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0170479a-87ab-464c-accc-a9e8f8fb4a1d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:45.603661 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.603640 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0170479a-87ab-464c-accc-a9e8f8fb4a1d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:45.625177 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.625147 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbdw\" (UniqueName: \"kubernetes.io/projected/0170479a-87ab-464c-accc-a9e8f8fb4a1d-kube-api-access-lxbdw\") pod \"payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:45.731357 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:45.731327 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:46.007172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:46.007143 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" event={"ID":"c2e6223e-5e32-423e-8dec-841ed7fa45ee","Type":"ContainerStarted","Data":"44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573"} Nov 03 23:13:46.007631 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:46.007196 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" event={"ID":"c2e6223e-5e32-423e-8dec-841ed7fa45ee","Type":"ContainerStarted","Data":"58f46bf300bdacc898fe78eb1e60b22d2ce158c380ed720cc2ff8c1f3b706450"} Nov 03 23:13:46.007631 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:46.007238 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:46.036782 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:46.036753 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss"] Nov 03 23:13:46.037303 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:13:46.037276 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0170479a_87ab_464c_accc_a9e8f8fb4a1d.slice/crio-ba048595c46d1574510c3010a11224441d7becadb176d2c26cd01e3105262639 WatchSource:0}: Error finding container ba048595c46d1574510c3010a11224441d7becadb176d2c26cd01e3105262639: Status 404 returned error can't find the container with id ba048595c46d1574510c3010a11224441d7becadb176d2c26cd01e3105262639 Nov 03 23:13:46.047044 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:46.046972 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" podStartSLOduration=1.681966364 podStartE2EDuration="2.046954651s" totalImagesPullingTime="364.988287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:13:46.046158553 +0000 UTC m=+3647.475494545" watchObservedRunningTime="2026-11-03 23:13:46.046954651 +0000 UTC m=+3647.476290642" Nov 03 23:13:47.015188 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:47.015135 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" event={"ID":"0170479a-87ab-464c-accc-a9e8f8fb4a1d","Type":"ContainerStarted","Data":"aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f"} Nov 03 23:13:47.015690 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:47.015210 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" event={"ID":"0170479a-87ab-464c-accc-a9e8f8fb4a1d","Type":"ContainerStarted","Data":"ba048595c46d1574510c3010a11224441d7becadb176d2c26cd01e3105262639"} Nov 03 23:13:47.015690 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:47.015478 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:13:47.058993 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:47.058941 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" podStartSLOduration=2.058928847 podStartE2EDuration="2.058928847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:13:47.055285656 +0000 UTC m=+3648.484621648" watchObservedRunningTime="2026-11-03 23:13:47.058928847 +0000 UTC m=+3648.488264839" Nov 03 23:13:52.021126 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:52.021094 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:13:58.025807 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:13:58.025777 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:14:02.185541 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:14:02.185503 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:14:07.651452 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:07.651420 4625 scope.go:122] "RemoveContainer" containerID="8f2d736972405141a210d7e442b2ef83084929d574d8ca17ec4eebb4247d63e6" Nov 03 23:14:07.661053 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:07.661026 4625 scope.go:122] "RemoveContainer" containerID="9c96bf8b660e405875a251ab88b2d708d471130e70c296bf4688f6d00aac46bd" Nov 03 23:14:15.029591 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:15.029555 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696"] Nov 03 23:14:15.040366 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:15.040340 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sj696"] Nov 03 23:14:16.727473 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:16.727438 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4330f85-82ce-45df-a0b3-ae1e3feee0c1" path="/var/lib/kubelet/pods/f4330f85-82ce-45df-a0b3-ae1e3feee0c1/volumes" Nov 03 23:14:32.185092 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:14:32.185056 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:14:35.042138 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:35.042103 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg"] Nov 03 23:14:35.053227 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:35.053192 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x6lqg"] Nov 03 23:14:36.726418 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:36.726388 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43060d43-6985-4fbe-a15a-c4122054796d" path="/var/lib/kubelet/pods/43060d43-6985-4fbe-a15a-c4122054796d/volumes" Nov 03 23:14:46.038739 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:46.038674 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd"] Nov 03 23:14:46.049207 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:46.049177 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zvzmd"] Nov 03 23:14:46.727018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:46.726986 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a627397-085a-4dfe-8474-be8a251076e0" path="/var/lib/kubelet/pods/4a627397-085a-4dfe-8474-be8a251076e0/volumes" Nov 03 23:14:54.316483 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:54.316444 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss"] Nov 03 23:14:55.376594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.376560 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h"] Nov 03 23:14:55.377517 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.377482 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" podUID="c2e6223e-5e32-423e-8dec-841ed7fa45ee" containerName="maas-api" containerID="cri-o://44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573" gracePeriod=30 Nov 03 23:14:55.881951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.881927 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:14:55.972221 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.972141 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e6223e-5e32-423e-8dec-841ed7fa45ee-kube-api-access-jh9sr\" (UniqueName: \"kubernetes.io/projected/c2e6223e-5e32-423e-8dec-841ed7fa45ee-kube-api-access-jh9sr\") pod \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " Nov 03 23:14:55.972221 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.972201 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-maas-api-tls\") pod \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " Nov 03 23:14:55.972453 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.972281 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e6223e-5e32-423e-8dec-841ed7fa45ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e6223e-5e32-423e-8dec-841ed7fa45ee-openshift-service-ca\") pod \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " Nov 03 23:14:55.972453 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.972346 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-metrics-certs\") pod \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\" (UID: \"c2e6223e-5e32-423e-8dec-841ed7fa45ee\") " Nov 03 23:14:55.972681 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.972652 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e6223e-5e32-423e-8dec-841ed7fa45ee-openshift-service-ca" pod "c2e6223e-5e32-423e-8dec-841ed7fa45ee" (UID: "c2e6223e-5e32-423e-8dec-841ed7fa45ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:55.974335 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.974312 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-metrics-certs" pod "c2e6223e-5e32-423e-8dec-841ed7fa45ee" (UID: "c2e6223e-5e32-423e-8dec-841ed7fa45ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:55.974778 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.974761 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-maas-api-tls" pod "c2e6223e-5e32-423e-8dec-841ed7fa45ee" (UID: "c2e6223e-5e32-423e-8dec-841ed7fa45ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:55.974869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:55.974799 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e6223e-5e32-423e-8dec-841ed7fa45ee-kube-api-access-jh9sr" pod "c2e6223e-5e32-423e-8dec-841ed7fa45ee" (UID: "c2e6223e-5e32-423e-8dec-841ed7fa45ee"). InnerVolumeSpecName "kube-api-access-jh9sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:56.084997 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.084960 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9sr\" (UniqueName: \"kubernetes.io/projected/c2e6223e-5e32-423e-8dec-841ed7fa45ee-kube-api-access-jh9sr\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:14:56.084997 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.084996 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:14:56.085262 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.085009 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e6223e-5e32-423e-8dec-841ed7fa45ee-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:14:56.085262 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.085020 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e6223e-5e32-423e-8dec-841ed7fa45ee-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:14:56.489856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.489821 4625 generic.go:356] "Generic (PLEG): container finished" podID="c2e6223e-5e32-423e-8dec-841ed7fa45ee" containerID="44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573" exitCode=0 Nov 03 23:14:56.490367 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.489868 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" event={"ID":"c2e6223e-5e32-423e-8dec-841ed7fa45ee","Type":"ContainerDied","Data":"44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573"} Nov 03 23:14:56.490367 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.489873 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" Nov 03 23:14:56.490367 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.489907 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h" event={"ID":"c2e6223e-5e32-423e-8dec-841ed7fa45ee","Type":"ContainerDied","Data":"58f46bf300bdacc898fe78eb1e60b22d2ce158c380ed720cc2ff8c1f3b706450"} Nov 03 23:14:56.490367 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.489927 4625 scope.go:122] "RemoveContainer" containerID="44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573" Nov 03 23:14:56.500640 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.500622 4625 scope.go:122] "RemoveContainer" containerID="44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573" Nov 03 23:14:56.500944 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:14:56.500924 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573\": container with ID starting with 44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573 not found: ID does not exist" containerID="44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573" Nov 03 23:14:56.500991 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.500954 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573"} err="failed to get container status \"44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573\": rpc error: code = NotFound desc = could not find container \"44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573\": container with ID starting with 44b53b02cf577985817eccc7581a7efe6d73b3ffd8e90dd9531188811c867573 not found: ID does not exist" Nov 03 23:14:56.541322 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.541291 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h"] Nov 03 23:14:56.550607 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.550585 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d5590b-694745c5df-tc94h"] Nov 03 23:14:56.726899 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:56.726866 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e6223e-5e32-423e-8dec-841ed7fa45ee" path="/var/lib/kubelet/pods/c2e6223e-5e32-423e-8dec-841ed7fa45ee/volumes" Nov 03 23:14:59.317201 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:14:59.317154 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" podUID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" containerName="payload-processing" containerID="cri-o://aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f" gracePeriod=25 Nov 03 23:15:00.137073 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.137036 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6"] Nov 03 23:15:00.137513 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.137499 4625 cpu_manager.go:414] "containerMap: removing container" podUID="c2e6223e-5e32-423e-8dec-841ed7fa45ee" containerName="maas-api" Nov 03 23:15:00.137565 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.137514 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e6223e-5e32-423e-8dec-841ed7fa45ee" containerName="maas-api" Nov 03 23:15:00.137608 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.137580 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e6223e-5e32-423e-8dec-841ed7fa45ee" containerName="maas-api" Nov 03 23:15:00.142128 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.142108 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.160244 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.160211 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6"] Nov 03 23:15:00.167836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.167808 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:15:00.329569 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.329537 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-wgdh6\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.330023 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.329585 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncqp\" (UniqueName: \"kubernetes.io/projected/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-kube-api-access-pncqp\") pod \"maas-api-key-cleanup-29895795-wgdh6\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.430818 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.430705 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncqp\" (UniqueName: \"kubernetes.io/projected/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-kube-api-access-pncqp\") pod \"maas-api-key-cleanup-29895795-wgdh6\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.431005 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.430868 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-wgdh6\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.431466 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.431444 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-wgdh6\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.452951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.452924 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncqp\" (UniqueName: \"kubernetes.io/projected/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-kube-api-access-pncqp\") pod \"maas-api-key-cleanup-29895795-wgdh6\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.455690 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.455669 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:00.762566 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:00.762526 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6"] Nov 03 23:15:00.765403 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:15:00.765375 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe33c9d_9c6b_42bc_9091_61f2fe66b8d3.slice/crio-0d1f963759d9367184dd8960a7cb2095011695a61cc159ba1c0e7be79dcf0075 WatchSource:0}: Error finding container 0d1f963759d9367184dd8960a7cb2095011695a61cc159ba1c0e7be79dcf0075: Status 404 returned error can't find the container with id 0d1f963759d9367184dd8960a7cb2095011695a61cc159ba1c0e7be79dcf0075 Nov 03 23:15:01.547157 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:01.547124 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" event={"ID":"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3","Type":"ContainerStarted","Data":"5ee468e665dbcfc8ef75b854191422285e7c9e303aea6ce1b71ce99225b1a7f7"} Nov 03 23:15:01.547596 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:01.547184 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" event={"ID":"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3","Type":"ContainerStarted","Data":"0d1f963759d9367184dd8960a7cb2095011695a61cc159ba1c0e7be79dcf0075"} Nov 03 23:15:01.587585 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:01.587525 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" podStartSLOduration=1.587510424 podStartE2EDuration="1.587510424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:01.584527832 +0000 UTC m=+3723.013863825" watchObservedRunningTime="2026-11-03 23:15:01.587510424 +0000 UTC m=+3723.016846416" Nov 03 23:15:02.185290 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:15:02.185246 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:15:02.556136 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:02.556055 4625 generic.go:356] "Generic (PLEG): container finished" podID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerID="5ee468e665dbcfc8ef75b854191422285e7c9e303aea6ce1b71ce99225b1a7f7" exitCode=60 Nov 03 23:15:02.556515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:02.556135 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" event={"ID":"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3","Type":"ContainerDied","Data":"5ee468e665dbcfc8ef75b854191422285e7c9e303aea6ce1b71ce99225b1a7f7"} Nov 03 23:15:02.556515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:02.556341 4625 scope.go:122] "RemoveContainer" containerID="5ee468e665dbcfc8ef75b854191422285e7c9e303aea6ce1b71ce99225b1a7f7" Nov 03 23:15:03.566664 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:03.566628 4625 generic.go:356] "Generic (PLEG): container finished" podID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerID="5e5d81ca289d7b223d1d36682949fb5b21702615246777671b8ffc6984367c15" exitCode=60 Nov 03 23:15:03.567192 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:03.566739 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" event={"ID":"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3","Type":"ContainerDied","Data":"5e5d81ca289d7b223d1d36682949fb5b21702615246777671b8ffc6984367c15"} Nov 03 23:15:03.567192 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:03.566818 4625 scope.go:122] "RemoveContainer" containerID="5ee468e665dbcfc8ef75b854191422285e7c9e303aea6ce1b71ce99225b1a7f7" Nov 03 23:15:03.567192 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:03.567074 4625 scope.go:122] "RemoveContainer" containerID="5e5d81ca289d7b223d1d36682949fb5b21702615246777671b8ffc6984367c15" Nov 03 23:15:03.567418 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:15:03.567388 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-wgdh6_odh-ai-gateway-infra(4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" Nov 03 23:15:04.202704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:04.202665 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5"] Nov 03 23:15:04.205363 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:04.205338 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7wm5"] Nov 03 23:15:04.576301 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:04.576225 4625 scope.go:122] "RemoveContainer" containerID="5e5d81ca289d7b223d1d36682949fb5b21702615246777671b8ffc6984367c15" Nov 03 23:15:04.576687 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:15:04.576516 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-wgdh6_odh-ai-gateway-infra(4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" Nov 03 23:15:04.727902 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:04.727762 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ecc126-1657-4dd7-b5d6-e4d3055842ec" path="/var/lib/kubelet/pods/b7ecc126-1657-4dd7-b5d6-e4d3055842ec/volumes" Nov 03 23:15:07.692650 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:07.692616 4625 scope.go:122] "RemoveContainer" containerID="2b71e5b296913984d36480c8d0c6eb4e8db5a02c592bf4cf0546032ba853700e" Nov 03 23:15:08.024850 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:08.024762 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" podUID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:15:14.029961 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:14.029875 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp"] Nov 03 23:15:14.041308 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:14.041271 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfrfp"] Nov 03 23:15:14.727298 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:14.727256 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f4a165-bc87-4ed1-9d44-920f128d4a77" path="/var/lib/kubelet/pods/e2f4a165-bc87-4ed1-9d44-920f128d4a77/volumes" Nov 03 23:15:15.720134 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:15.720102 4625 scope.go:122] "RemoveContainer" containerID="5e5d81ca289d7b223d1d36682949fb5b21702615246777671b8ffc6984367c15" Nov 03 23:15:16.663354 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:16.663317 4625 generic.go:356] "Generic (PLEG): container finished" podID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerID="bba0485269b29248fd1fb57e02e6290b718bab97f54e28ee1abc20d935f72099" exitCode=60 Nov 03 23:15:16.663534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:16.663387 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" event={"ID":"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3","Type":"ContainerDied","Data":"bba0485269b29248fd1fb57e02e6290b718bab97f54e28ee1abc20d935f72099"} Nov 03 23:15:16.663534 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:16.663449 4625 scope.go:122] "RemoveContainer" containerID="5e5d81ca289d7b223d1d36682949fb5b21702615246777671b8ffc6984367c15" Nov 03 23:15:16.663842 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:16.663825 4625 scope.go:122] "RemoveContainer" containerID="bba0485269b29248fd1fb57e02e6290b718bab97f54e28ee1abc20d935f72099" Nov 03 23:15:16.664131 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:15:16.664103 4625 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-wgdh6_odh-ai-gateway-infra(4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" Nov 03 23:15:17.714846 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:17.714810 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6"] Nov 03 23:15:18.029799 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.029545 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" podUID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:15:18.167750 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.166368 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:18.229084 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.229050 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-maas-api-service-ca\") pod \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " Nov 03 23:15:18.229323 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.229310 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-kube-api-access-pncqp\" (UniqueName: \"kubernetes.io/projected/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-kube-api-access-pncqp\") pod \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\" (UID: \"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3\") " Nov 03 23:15:18.229480 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.229449 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-maas-api-service-ca" pod "4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" (UID: "4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:18.229857 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.229837 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-maas-api-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:15:18.247895 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.232776 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-kube-api-access-pncqp" pod "4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" (UID: "4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3"). InnerVolumeSpecName "kube-api-access-pncqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:18.356667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.356420 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncqp\" (UniqueName: \"kubernetes.io/projected/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3-kube-api-access-pncqp\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:15:18.711252 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.711156 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" event={"ID":"4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3","Type":"ContainerDied","Data":"0d1f963759d9367184dd8960a7cb2095011695a61cc159ba1c0e7be79dcf0075"} Nov 03 23:15:18.711252 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.711223 4625 scope.go:122] "RemoveContainer" containerID="bba0485269b29248fd1fb57e02e6290b718bab97f54e28ee1abc20d935f72099" Nov 03 23:15:18.711514 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.711170 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6" Nov 03 23:15:18.769794 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.769761 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6"] Nov 03 23:15:18.773505 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:18.773476 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-wgdh6"] Nov 03 23:15:19.599378 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.599351 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:15:19.669780 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.669734 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0170479a-87ab-464c-accc-a9e8f8fb4a1d-kube-api-access-lxbdw\" (UniqueName: \"kubernetes.io/projected/0170479a-87ab-464c-accc-a9e8f8fb4a1d-kube-api-access-lxbdw\") pod \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " Nov 03 23:15:19.669964 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.669809 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0170479a-87ab-464c-accc-a9e8f8fb4a1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0170479a-87ab-464c-accc-a9e8f8fb4a1d-plugins-config-volume\") pod \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\" (UID: \"0170479a-87ab-464c-accc-a9e8f8fb4a1d\") " Nov 03 23:15:19.670213 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.670185 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0170479a-87ab-464c-accc-a9e8f8fb4a1d-plugins-config-volume" pod "0170479a-87ab-464c-accc-a9e8f8fb4a1d" (UID: "0170479a-87ab-464c-accc-a9e8f8fb4a1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:19.671834 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.671806 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0170479a-87ab-464c-accc-a9e8f8fb4a1d-kube-api-access-lxbdw" pod "0170479a-87ab-464c-accc-a9e8f8fb4a1d" (UID: "0170479a-87ab-464c-accc-a9e8f8fb4a1d"). InnerVolumeSpecName "kube-api-access-lxbdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:19.737920 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.737883 4625 generic.go:356] "Generic (PLEG): container finished" podID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" containerID="aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f" exitCode=0 Nov 03 23:15:19.738214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.737955 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" Nov 03 23:15:19.739138 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.737975 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" event={"ID":"0170479a-87ab-464c-accc-a9e8f8fb4a1d","Type":"ContainerDied","Data":"aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f"} Nov 03 23:15:19.739273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.739182 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss" event={"ID":"0170479a-87ab-464c-accc-a9e8f8fb4a1d","Type":"ContainerDied","Data":"ba048595c46d1574510c3010a11224441d7becadb176d2c26cd01e3105262639"} Nov 03 23:15:19.739273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.739240 4625 scope.go:122] "RemoveContainer" containerID="aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f" Nov 03 23:15:19.763118 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.763084 4625 scope.go:122] "RemoveContainer" containerID="aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f" Nov 03 23:15:19.763506 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:15:19.763476 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f\": container with ID starting with aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f not found: ID does not exist" containerID="aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f" Nov 03 23:15:19.763598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.763520 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f"} err="failed to get container status \"aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f\": rpc error: code = NotFound desc = could not find container \"aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f\": container with ID starting with aca223160b3f7f6b6dcc7f20545731fb83abcaa00bbfbe7c6acb052561d3596f not found: ID does not exist" Nov 03 23:15:19.770514 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.770490 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbdw\" (UniqueName: \"kubernetes.io/projected/0170479a-87ab-464c-accc-a9e8f8fb4a1d-kube-api-access-lxbdw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:15:19.770514 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.770514 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0170479a-87ab-464c-accc-a9e8f8fb4a1d-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:15:19.790839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.790804 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss"] Nov 03 23:15:19.795455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:19.795425 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d5590b-75b6cb5f9ll2ss"] Nov 03 23:15:20.726578 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:20.726536 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" path="/var/lib/kubelet/pods/0170479a-87ab-464c-accc-a9e8f8fb4a1d/volumes" Nov 03 23:15:20.727049 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:20.727028 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" path="/var/lib/kubelet/pods/4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3/volumes" Nov 03 23:15:25.028186 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:25.028146 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp"] Nov 03 23:15:25.039477 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:25.039436 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-58snp"] Nov 03 23:15:26.727062 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:26.727019 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b493fce4-3699-4660-b793-111e316e9cc1" path="/var/lib/kubelet/pods/b493fce4-3699-4660-b793-111e316e9cc1/volumes" Nov 03 23:15:28.343653 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.343614 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7"] Nov 03 23:15:28.344143 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.343866 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" podUID="d48f2e0c-8877-44f4-8c75-948fcb5f03d0" containerName="manager" containerID="cri-o://a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3" gracePeriod=10 Nov 03 23:15:28.611537 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.611507 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 23:15:28.659686 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.659652 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-extensions-socket-volume\") pod \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " Nov 03 23:15:28.659912 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.659733 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-kube-api-access-pms5d\" (UniqueName: \"kubernetes.io/projected/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-kube-api-access-pms5d\") pod \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\" (UID: \"d48f2e0c-8877-44f4-8c75-948fcb5f03d0\") " Nov 03 23:15:28.660122 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.660098 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-extensions-socket-volume" pod "d48f2e0c-8877-44f4-8c75-948fcb5f03d0" (UID: "d48f2e0c-8877-44f4-8c75-948fcb5f03d0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:28.662775 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.662750 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-kube-api-access-pms5d" pod "d48f2e0c-8877-44f4-8c75-948fcb5f03d0" (UID: "d48f2e0c-8877-44f4-8c75-948fcb5f03d0"). InnerVolumeSpecName "kube-api-access-pms5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:28.761273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.761238 4625 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-extensions-socket-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:15:28.761494 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.761355 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pms5d\" (UniqueName: \"kubernetes.io/projected/d48f2e0c-8877-44f4-8c75-948fcb5f03d0-kube-api-access-pms5d\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:15:28.797378 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.797339 4625 generic.go:356] "Generic (PLEG): container finished" podID="d48f2e0c-8877-44f4-8c75-948fcb5f03d0" containerID="a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3" exitCode=0 Nov 03 23:15:28.797572 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.797387 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" Nov 03 23:15:28.797572 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.797420 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" event={"ID":"d48f2e0c-8877-44f4-8c75-948fcb5f03d0","Type":"ContainerDied","Data":"a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3"} Nov 03 23:15:28.797572 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.797477 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7" event={"ID":"d48f2e0c-8877-44f4-8c75-948fcb5f03d0","Type":"ContainerDied","Data":"c0aa6f2c357adb1b0b7023e8388f9e878ea33e908d0f0bdf8498298b64c4841f"} Nov 03 23:15:28.797572 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.797501 4625 scope.go:122] "RemoveContainer" containerID="a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3" Nov 03 23:15:28.810337 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.810316 4625 scope.go:122] "RemoveContainer" containerID="a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3" Nov 03 23:15:28.810679 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:15:28.810648 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3\": container with ID starting with a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3 not found: ID does not exist" containerID="a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3" Nov 03 23:15:28.810814 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.810685 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3"} err="failed to get container status \"a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3\": rpc error: code = NotFound desc = could not find container \"a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3\": container with ID starting with a3da2f3ddb478962bc711c03b5ca862cbd64d03cabe082971549937c80bf98e3 not found: ID does not exist" Nov 03 23:15:28.847702 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.847474 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7"] Nov 03 23:15:28.851014 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:28.850976 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rfdx7"] Nov 03 23:15:30.726223 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:30.726187 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48f2e0c-8877-44f4-8c75-948fcb5f03d0" path="/var/lib/kubelet/pods/d48f2e0c-8877-44f4-8c75-948fcb5f03d0/volumes" Nov 03 23:15:32.186354 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:15:32.186309 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:15:46.027782 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:46.027743 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm"] Nov 03 23:15:46.038825 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:46.038791 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ck7tm"] Nov 03 23:15:46.730147 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:15:46.730115 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205c43a6-7f74-45d7-8f38-27f05578b5ba" path="/var/lib/kubelet/pods/205c43a6-7f74-45d7-8f38-27f05578b5ba/volumes" Nov 03 23:16:02.185185 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:16:02.185143 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:16:07.724265 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:07.724228 4625 scope.go:122] "RemoveContainer" containerID="d0c5670ae13084b48d0a306c7e5abdb3009f20f42439e4afb897fcfdee0d0c5f" Nov 03 23:16:07.733333 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:07.733307 4625 scope.go:122] "RemoveContainer" containerID="468a5fa4e1fbb30dad882a589c61b3ad6029b804024a785dbed806a32e90c108" Nov 03 23:16:07.741496 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:07.741474 4625 scope.go:122] "RemoveContainer" containerID="d97bb797ef7c8bd355ebd90192a21dd82fbdae9c442040dd4637444748753f0b" Nov 03 23:16:08.029211 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:08.029125 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz"] Nov 03 23:16:08.040111 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:08.040080 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dczvz"] Nov 03 23:16:08.726185 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:08.726154 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539fdf04-bf7e-47cc-8374-6d60692dd540" path="/var/lib/kubelet/pods/539fdf04-bf7e-47cc-8374-6d60692dd540/volumes" Nov 03 23:16:30.028623 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:30.028588 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2"] Nov 03 23:16:30.039310 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:30.039281 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w76s2"] Nov 03 23:16:30.726568 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:30.726535 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4483708-7c3b-436c-837a-359bbaa4ffd1" path="/var/lib/kubelet/pods/a4483708-7c3b-436c-837a-359bbaa4ffd1/volumes" Nov 03 23:16:31.497274 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497234 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd"] Nov 03 23:16:31.497709 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497693 4625 cpu_manager.go:414] "containerMap: removing container" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497709 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497705 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497709 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497728 4625 cpu_manager.go:414] "containerMap: removing container" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497709 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497733 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497709 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497743 4625 cpu_manager.go:414] "containerMap: removing container" podUID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" containerName="payload-processing" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497749 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" containerName="payload-processing" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497760 4625 cpu_manager.go:414] "containerMap: removing container" podUID="d48f2e0c-8877-44f4-8c75-948fcb5f03d0" containerName="manager" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497767 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48f2e0c-8877-44f4-8c75-948fcb5f03d0" containerName="manager" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497783 4625 cpu_manager.go:414] "containerMap: removing container" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497788 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497878 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48f2e0c-8877-44f4-8c75-948fcb5f03d0" containerName="manager" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497888 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497901 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:31.497987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.497906 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="0170479a-87ab-464c-accc-a9e8f8fb4a1d" containerName="payload-processing" Nov 03 23:16:31.503086 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.503066 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:31.518340 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.518299 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd"] Nov 03 23:16:31.525919 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.525883 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zc882\"" Nov 03 23:16:31.682881 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.682841 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9c3eed8-d000-4186-a005-25d95469cb1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rhpcd\" (UID: \"c9c3eed8-d000-4186-a005-25d95469cb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:31.683114 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.683056 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzj59\" (UniqueName: \"kubernetes.io/projected/c9c3eed8-d000-4186-a005-25d95469cb1f-kube-api-access-bzj59\") pod \"kuadrant-operator-controller-manager-678d87766f-rhpcd\" (UID: \"c9c3eed8-d000-4186-a005-25d95469cb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:31.784588 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.784490 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzj59\" (UniqueName: \"kubernetes.io/projected/c9c3eed8-d000-4186-a005-25d95469cb1f-kube-api-access-bzj59\") pod \"kuadrant-operator-controller-manager-678d87766f-rhpcd\" (UID: \"c9c3eed8-d000-4186-a005-25d95469cb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:31.784588 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.784543 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9c3eed8-d000-4186-a005-25d95469cb1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rhpcd\" (UID: \"c9c3eed8-d000-4186-a005-25d95469cb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:31.785006 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.784984 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9c3eed8-d000-4186-a005-25d95469cb1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rhpcd\" (UID: \"c9c3eed8-d000-4186-a005-25d95469cb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:31.807104 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.807075 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzj59\" (UniqueName: \"kubernetes.io/projected/c9c3eed8-d000-4186-a005-25d95469cb1f-kube-api-access-bzj59\") pod \"kuadrant-operator-controller-manager-678d87766f-rhpcd\" (UID: \"c9c3eed8-d000-4186-a005-25d95469cb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:31.819923 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:31.819899 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:32.155736 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:32.155683 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd"] Nov 03 23:16:32.161270 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:16:32.161236 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c3eed8_d000_4186_a005_25d95469cb1f.slice/crio-c750d7324abeff105d746250f4451cbf48d2860a09f0fb783ab0751edc0f5421 WatchSource:0}: Error finding container c750d7324abeff105d746250f4451cbf48d2860a09f0fb783ab0751edc0f5421: Status 404 returned error can't find the container with id c750d7324abeff105d746250f4451cbf48d2860a09f0fb783ab0751edc0f5421 Nov 03 23:16:32.187494 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:32.186192 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" event={"ID":"c9c3eed8-d000-4186-a005-25d95469cb1f","Type":"ContainerStarted","Data":"c750d7324abeff105d746250f4451cbf48d2860a09f0fb783ab0751edc0f5421"} Nov 03 23:16:32.194389 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:16:32.194354 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:16:33.193659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:33.193611 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" event={"ID":"c9c3eed8-d000-4186-a005-25d95469cb1f","Type":"ContainerStarted","Data":"383fb5f31203378f23be1a711b4a1e8c5c61161dd690cbf6c3e89515a179c8aa"} Nov 03 23:16:33.194172 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:33.193703 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:33.257385 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:33.257331 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" podStartSLOduration=2.257319992 podStartE2EDuration="2.257319992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:16:33.245825838 +0000 UTC m=+3814.675161827" watchObservedRunningTime="2026-11-03 23:16:33.257319992 +0000 UTC m=+3814.686655985" Nov 03 23:16:44.201982 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:44.201903 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhpcd" Nov 03 23:16:47.029982 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:47.029945 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx"] Nov 03 23:16:47.040117 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:47.040088 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fzkrx"] Nov 03 23:16:48.729510 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:48.729472 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674c0ae7-5f93-4c7b-9b60-43bb0643e018" path="/var/lib/kubelet/pods/674c0ae7-5f93-4c7b-9b60-43bb0643e018/volumes" Nov 03 23:16:53.339130 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.339086 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz"] Nov 03 23:16:53.339646 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.339611 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe33c9d-9c6b-42bc-9091-61f2fe66b8d3" containerName="cleanup" Nov 03 23:16:53.342895 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.342873 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.370049 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.370016 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:16:53.370203 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.370072 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:16:53.374278 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.374247 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz"] Nov 03 23:16:53.423453 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.423413 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.423641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.423469 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcw4\" (UniqueName: \"kubernetes.io/projected/76477a57-ab89-4bc8-b993-044b447788b4-kube-api-access-npcw4\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.423641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.423571 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.525202 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.525164 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.525405 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.525218 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcw4\" (UniqueName: \"kubernetes.io/projected/76477a57-ab89-4bc8-b993-044b447788b4-kube-api-access-npcw4\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.525405 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.525248 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.526031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.526004 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.527730 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.527691 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.547533 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.547504 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcw4\" (UniqueName: \"kubernetes.io/projected/76477a57-ab89-4bc8-b993-044b447788b4-kube-api-access-npcw4\") pod \"maas-api-revoke-sub-d576e68b-vv4hz\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.661540 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.661455 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:53.968296 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:53.968130 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz"] Nov 03 23:16:53.970651 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:16:53.970621 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76477a57_ab89_4bc8_b993_044b447788b4.slice/crio-fdbb701861bdd33c1cac5c589d40f1b3eeecd455c060d61c3e2464eaec9a1909 WatchSource:0}: Error finding container fdbb701861bdd33c1cac5c589d40f1b3eeecd455c060d61c3e2464eaec9a1909: Status 404 returned error can't find the container with id fdbb701861bdd33c1cac5c589d40f1b3eeecd455c060d61c3e2464eaec9a1909 Nov 03 23:16:54.331687 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:54.331642 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" event={"ID":"76477a57-ab89-4bc8-b993-044b447788b4","Type":"ContainerStarted","Data":"3bab09998e51daed7517126b95a85883fbaa8de1c6dca32fe097298b2043e6df"} Nov 03 23:16:54.331895 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:54.331737 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" event={"ID":"76477a57-ab89-4bc8-b993-044b447788b4","Type":"ContainerStarted","Data":"fdbb701861bdd33c1cac5c589d40f1b3eeecd455c060d61c3e2464eaec9a1909"} Nov 03 23:16:54.368777 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:54.368725 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" podStartSLOduration=1.3687006 podStartE2EDuration="1.3687006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:16:54.3685544 +0000 UTC m=+3835.797890392" watchObservedRunningTime="2026-11-03 23:16:54.3687006 +0000 UTC m=+3835.798036590" Nov 03 23:16:55.339444 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:55.339411 4625 generic.go:356] "Generic (PLEG): container finished" podID="76477a57-ab89-4bc8-b993-044b447788b4" containerID="3bab09998e51daed7517126b95a85883fbaa8de1c6dca32fe097298b2043e6df" exitCode=0 Nov 03 23:16:55.339600 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:55.339490 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" event={"ID":"76477a57-ab89-4bc8-b993-044b447788b4","Type":"ContainerDied","Data":"3bab09998e51daed7517126b95a85883fbaa8de1c6dca32fe097298b2043e6df"} Nov 03 23:16:56.477908 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.477885 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:56.552966 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.552939 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76477a57-ab89-4bc8-b993-044b447788b4-kube-api-access-npcw4\" (UniqueName: \"kubernetes.io/projected/76477a57-ab89-4bc8-b993-044b447788b4-kube-api-access-npcw4\") pod \"76477a57-ab89-4bc8-b993-044b447788b4\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " Nov 03 23:16:56.553153 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.553078 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-xks-ca\") pod \"76477a57-ab89-4bc8-b993-044b447788b4\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " Nov 03 23:16:56.553153 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.553137 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-openshift-ca\") pod \"76477a57-ab89-4bc8-b993-044b447788b4\" (UID: \"76477a57-ab89-4bc8-b993-044b447788b4\") " Nov 03 23:16:56.553537 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.553500 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-openshift-ca" pod "76477a57-ab89-4bc8-b993-044b447788b4" (UID: "76477a57-ab89-4bc8-b993-044b447788b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:16:56.555081 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.555058 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76477a57-ab89-4bc8-b993-044b447788b4-kube-api-access-npcw4" pod "76477a57-ab89-4bc8-b993-044b447788b4" (UID: "76477a57-ab89-4bc8-b993-044b447788b4"). InnerVolumeSpecName "kube-api-access-npcw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:16:56.555261 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.555245 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-xks-ca" pod "76477a57-ab89-4bc8-b993-044b447788b4" (UID: "76477a57-ab89-4bc8-b993-044b447788b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:16:56.654699 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.654623 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:16:56.654699 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.654651 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76477a57-ab89-4bc8-b993-044b447788b4-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:16:56.654699 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:56.654662 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npcw4\" (UniqueName: \"kubernetes.io/projected/76477a57-ab89-4bc8-b993-044b447788b4-kube-api-access-npcw4\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:16:57.352659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:57.352618 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" event={"ID":"76477a57-ab89-4bc8-b993-044b447788b4","Type":"ContainerDied","Data":"fdbb701861bdd33c1cac5c589d40f1b3eeecd455c060d61c3e2464eaec9a1909"} Nov 03 23:16:57.352659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:57.352659 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz" Nov 03 23:16:57.352939 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:16:57.352696 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbb701861bdd33c1cac5c589d40f1b3eeecd455c060d61c3e2464eaec9a1909" Nov 03 23:17:02.185218 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:17:02.185183 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:17:07.787264 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:07.787216 4625 scope.go:122] "RemoveContainer" containerID="e74fab42cca482f036bb0a181517ff7f558c6412f9c05ef7623b8b3613fb7bff" Nov 03 23:17:07.800439 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:07.800415 4625 scope.go:122] "RemoveContainer" containerID="37c0a6abbc1f0e3793c4ae504845646b2bbcdf8a91f330312d02b4531f0294d7" Nov 03 23:17:07.812609 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:07.812584 4625 scope.go:122] "RemoveContainer" containerID="1b97e8cdcbb60d005bbe3fb81420a56596262b25c262e081d7e104080dce1fd1" Nov 03 23:17:07.825835 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:07.825790 4625 scope.go:122] "RemoveContainer" containerID="2fb49c48be366f65d266cb28c49ce01b0831427f4939decc172762b779b574e8" Nov 03 23:17:14.445236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.445200 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh"] Nov 03 23:17:14.445728 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.445699 4625 cpu_manager.go:414] "containerMap: removing container" podUID="76477a57-ab89-4bc8-b993-044b447788b4" containerName="revoke-keys" Nov 03 23:17:14.445788 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.445730 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76477a57-ab89-4bc8-b993-044b447788b4" containerName="revoke-keys" Nov 03 23:17:14.445827 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.445797 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="76477a57-ab89-4bc8-b993-044b447788b4" containerName="revoke-keys" Nov 03 23:17:14.450540 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.450514 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.466005 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.465797 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:17:14.477482 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.477450 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:17:14.480861 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.480836 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh"] Nov 03 23:17:14.571830 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.571795 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.572031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.571837 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.572031 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.571915 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsck\" (UniqueName: \"kubernetes.io/projected/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-kube-api-access-8dsck\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.673160 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.673128 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.673339 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.673173 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.673339 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.673297 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsck\" (UniqueName: \"kubernetes.io/projected/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-kube-api-access-8dsck\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.673934 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.673913 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.675511 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.675482 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.695315 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.695249 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsck\" (UniqueName: \"kubernetes.io/projected/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-kube-api-access-8dsck\") pod \"maas-api-revoke-sub-d2bc93bc-cq9kh\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:14.764569 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:14.764543 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:15.067369 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:15.067337 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh"] Nov 03 23:17:15.070353 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:17:15.070327 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda819ab71_48bd_45fd_ad42_ab4a9f9e85a7.slice/crio-c3f56eff08bcf622803bbf9ece5468822e7ff4f0a0ed62f6f10b66d06e52e7a9 WatchSource:0}: Error finding container c3f56eff08bcf622803bbf9ece5468822e7ff4f0a0ed62f6f10b66d06e52e7a9: Status 404 returned error can't find the container with id c3f56eff08bcf622803bbf9ece5468822e7ff4f0a0ed62f6f10b66d06e52e7a9 Nov 03 23:17:15.476434 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:15.476376 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" event={"ID":"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7","Type":"ContainerStarted","Data":"6e98e15850683afe9b715818ef20da2d60356104b320488caf11d3ff760c8b7d"} Nov 03 23:17:15.476870 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:15.476461 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" event={"ID":"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7","Type":"ContainerStarted","Data":"c3f56eff08bcf622803bbf9ece5468822e7ff4f0a0ed62f6f10b66d06e52e7a9"} Nov 03 23:17:15.514847 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:15.514795 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" podStartSLOduration=1.5147791640000001 podStartE2EDuration="1.514779164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:15.514076296 +0000 UTC m=+3856.943412286" watchObservedRunningTime="2026-11-03 23:17:15.514779164 +0000 UTC m=+3856.944115156" Nov 03 23:17:16.483530 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:16.483501 4625 generic.go:356] "Generic (PLEG): container finished" podID="a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" containerID="6e98e15850683afe9b715818ef20da2d60356104b320488caf11d3ff760c8b7d" exitCode=0 Nov 03 23:17:16.483956 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:16.483585 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" event={"ID":"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7","Type":"ContainerDied","Data":"6e98e15850683afe9b715818ef20da2d60356104b320488caf11d3ff760c8b7d"} Nov 03 23:17:17.627194 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.627166 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:17.700471 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.700432 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-openshift-ca\") pod \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " Nov 03 23:17:17.700659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.700526 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-kube-api-access-8dsck\" (UniqueName: \"kubernetes.io/projected/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-kube-api-access-8dsck\") pod \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " Nov 03 23:17:17.700659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.700565 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-xks-ca\") pod \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\" (UID: \"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7\") " Nov 03 23:17:17.700883 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.700857 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-openshift-ca" pod "a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" (UID: "a819ab71-48bd-45fd-ad42-ab4a9f9e85a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:17:17.702702 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.702671 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-xks-ca" pod "a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" (UID: "a819ab71-48bd-45fd-ad42-ab4a9f9e85a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:17:17.702890 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.702773 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-kube-api-access-8dsck" pod "a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" (UID: "a819ab71-48bd-45fd-ad42-ab4a9f9e85a7"). InnerVolumeSpecName "kube-api-access-8dsck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:17:17.801667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.801573 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:17:17.801667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.801614 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsck\" (UniqueName: \"kubernetes.io/projected/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-kube-api-access-8dsck\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:17:17.801667 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:17.801628 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:17:18.497339 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:18.497308 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" Nov 03 23:17:18.497614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:18.497313 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh" event={"ID":"a819ab71-48bd-45fd-ad42-ab4a9f9e85a7","Type":"ContainerDied","Data":"c3f56eff08bcf622803bbf9ece5468822e7ff4f0a0ed62f6f10b66d06e52e7a9"} Nov 03 23:17:18.497614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:18.497425 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f56eff08bcf622803bbf9ece5468822e7ff4f0a0ed62f6f10b66d06e52e7a9" Nov 03 23:17:32.067312 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.067273 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq"] Nov 03 23:17:32.067740 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.067708 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" containerName="revoke-keys" Nov 03 23:17:32.067794 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.067744 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" containerName="revoke-keys" Nov 03 23:17:32.067853 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.067844 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" containerName="revoke-keys" Nov 03 23:17:32.071374 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.071347 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.097161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.097115 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qn4p2\"" Nov 03 23:17:32.097521 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.096892 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-22c413-serving-cert\"" Nov 03 23:17:32.098697 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.096973 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:17:32.101968 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.101944 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq"] Nov 03 23:17:32.154897 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.154860 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc24v\" (UniqueName: \"kubernetes.io/projected/2d8471ab-9c53-48cd-9de4-9bd3918169de-kube-api-access-cc24v\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.155077 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.155013 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-metrics-certs\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.155077 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.155043 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.155171 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.155138 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8471ab-9c53-48cd-9de4-9bd3918169de-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.189755 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:17:32.189703 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:17:32.256585 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.256534 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-metrics-certs\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.256585 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.256590 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.256888 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.256653 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8471ab-9c53-48cd-9de4-9bd3918169de-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.256888 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.256784 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc24v\" (UniqueName: \"kubernetes.io/projected/2d8471ab-9c53-48cd-9de4-9bd3918169de-kube-api-access-cc24v\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.257476 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.257449 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8471ab-9c53-48cd-9de4-9bd3918169de-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.259312 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.259283 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-metrics-certs\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.259537 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.259510 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.279928 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.279897 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc24v\" (UniqueName: \"kubernetes.io/projected/2d8471ab-9c53-48cd-9de4-9bd3918169de-kube-api-access-cc24v\") pod \"maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.384913 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.384874 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:32.521282 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.521235 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578"] Nov 03 23:17:32.539904 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.539870 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:32.549872 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.549840 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578"] Nov 03 23:17:32.556298 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.556123 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-22c413\"" Nov 03 23:17:32.556298 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.556267 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7c7b11fe-dockercfg-kk8mz\"" Nov 03 23:17:32.661117 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.661039 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc337f7-728b-44a2-8650-be5e763686ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:32.661281 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.661225 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vth6n\" (UniqueName: \"kubernetes.io/projected/adc337f7-728b-44a2-8650-be5e763686ee-kube-api-access-vth6n\") pod \"payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:32.702077 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.702042 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq"] Nov 03 23:17:32.705395 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:17:32.705358 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8471ab_9c53_48cd_9de4_9bd3918169de.slice/crio-68e0e5ff323fc64094e170e35bccaf33ef1269df65e8b84d2dbd90abd5e9b299 WatchSource:0}: Error finding container 68e0e5ff323fc64094e170e35bccaf33ef1269df65e8b84d2dbd90abd5e9b299: Status 404 returned error can't find the container with id 68e0e5ff323fc64094e170e35bccaf33ef1269df65e8b84d2dbd90abd5e9b299 Nov 03 23:17:32.762086 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.762049 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vth6n\" (UniqueName: \"kubernetes.io/projected/adc337f7-728b-44a2-8650-be5e763686ee-kube-api-access-vth6n\") pod \"payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:32.762272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.762111 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc337f7-728b-44a2-8650-be5e763686ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:32.762840 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.762821 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc337f7-728b-44a2-8650-be5e763686ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:32.785189 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.785160 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vth6n\" (UniqueName: \"kubernetes.io/projected/adc337f7-728b-44a2-8650-be5e763686ee-kube-api-access-vth6n\") pod \"payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:32.866342 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:32.866304 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:33.174596 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.174561 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578"] Nov 03 23:17:33.177088 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:17:33.177065 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc337f7_728b_44a2_8650_be5e763686ee.slice/crio-0e5cdc51535fdfeb2b4e9a1c272a1a9774da616f535eff46050d7d0d6acc6d87 WatchSource:0}: Error finding container 0e5cdc51535fdfeb2b4e9a1c272a1a9774da616f535eff46050d7d0d6acc6d87: Status 404 returned error can't find the container with id 0e5cdc51535fdfeb2b4e9a1c272a1a9774da616f535eff46050d7d0d6acc6d87 Nov 03 23:17:33.601037 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.600991 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" event={"ID":"adc337f7-728b-44a2-8650-be5e763686ee","Type":"ContainerStarted","Data":"abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f"} Nov 03 23:17:33.601299 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.601093 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" event={"ID":"adc337f7-728b-44a2-8650-be5e763686ee","Type":"ContainerStarted","Data":"0e5cdc51535fdfeb2b4e9a1c272a1a9774da616f535eff46050d7d0d6acc6d87"} Nov 03 23:17:33.601381 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.601313 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:33.603065 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.603028 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" event={"ID":"2d8471ab-9c53-48cd-9de4-9bd3918169de","Type":"ContainerStarted","Data":"e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094"} Nov 03 23:17:33.603214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.603111 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" event={"ID":"2d8471ab-9c53-48cd-9de4-9bd3918169de","Type":"ContainerStarted","Data":"68e0e5ff323fc64094e170e35bccaf33ef1269df65e8b84d2dbd90abd5e9b299"} Nov 03 23:17:33.603214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.603155 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:33.647820 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.647752 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" podStartSLOduration=1.647729123 podStartE2EDuration="1.647729123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:33.642798616 +0000 UTC m=+3875.072134611" watchObservedRunningTime="2026-11-03 23:17:33.647729123 +0000 UTC m=+3875.077065110" Nov 03 23:17:33.688250 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:33.688184 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" podStartSLOduration=1.266187694 podStartE2EDuration="1.688166825s" totalImagesPullingTime="421.979131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:33.684989355 +0000 UTC m=+3875.114325349" watchObservedRunningTime="2026-11-03 23:17:33.688166825 +0000 UTC m=+3875.117502820" Nov 03 23:17:39.615256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:39.615226 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:17:44.612100 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:44.612074 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:17:51.039189 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:51.039151 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt"] Nov 03 23:17:51.050088 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:51.050053 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tqlqt"] Nov 03 23:17:52.727204 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:17:52.727169 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a40fe5-b941-466f-a0e9-fdad9af3eb9b" path="/var/lib/kubelet/pods/46a40fe5-b941-466f-a0e9-fdad9af3eb9b/volumes" Nov 03 23:18:02.028707 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:02.028662 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk"] Nov 03 23:18:02.039632 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:02.039601 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-stmlk"] Nov 03 23:18:02.184705 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:18:02.184669 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:18:02.727780 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:02.727744 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe5d8f9-b881-4d54-b40a-5be6c7f7540b" path="/var/lib/kubelet/pods/0fe5d8f9-b881-4d54-b40a-5be6c7f7540b/volumes" Nov 03 23:18:04.048449 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:04.048418 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:18:04.105931 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:04.105906 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:18:07.883976 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:07.883940 4625 scope.go:122] "RemoveContainer" containerID="f29f5875d50ef2819ef72c3b5577204dbadb764543f962a16095efce1b6e3f9e" Nov 03 23:18:07.893662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:07.893638 4625 scope.go:122] "RemoveContainer" containerID="8bb51dd80a10098678e6a07b74f99abecfaa94a354ceed19c9c3a185bfe6ff83" Nov 03 23:18:07.903299 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:07.903269 4625 scope.go:122] "RemoveContainer" containerID="db9c12f2f9efe1f44740def47c38b0c7ca5243ef43f3cfe44116ff22c66ec823" Nov 03 23:18:32.191052 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:18:32.190956 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:18:39.332350 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.332305 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq"] Nov 03 23:18:39.333024 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.332638 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" podUID="2d8471ab-9c53-48cd-9de4-9bd3918169de" containerName="maas-api" containerID="cri-o://e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094" gracePeriod=30 Nov 03 23:18:39.372835 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.372800 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578"] Nov 03 23:18:39.604504 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.604476 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:18:39.692836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.692798 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8471ab-9c53-48cd-9de4-9bd3918169de-kube-api-access-cc24v\" (UniqueName: \"kubernetes.io/projected/2d8471ab-9c53-48cd-9de4-9bd3918169de-kube-api-access-cc24v\") pod \"2d8471ab-9c53-48cd-9de4-9bd3918169de\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " Nov 03 23:18:39.693016 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.692881 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8471ab-9c53-48cd-9de4-9bd3918169de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8471ab-9c53-48cd-9de4-9bd3918169de-openshift-service-ca\") pod \"2d8471ab-9c53-48cd-9de4-9bd3918169de\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " Nov 03 23:18:39.693016 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.692983 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-maas-api-tls\") pod \"2d8471ab-9c53-48cd-9de4-9bd3918169de\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " Nov 03 23:18:39.693097 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.693014 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-metrics-certs\") pod \"2d8471ab-9c53-48cd-9de4-9bd3918169de\" (UID: \"2d8471ab-9c53-48cd-9de4-9bd3918169de\") " Nov 03 23:18:39.693289 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.693257 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8471ab-9c53-48cd-9de4-9bd3918169de-openshift-service-ca" pod "2d8471ab-9c53-48cd-9de4-9bd3918169de" (UID: "2d8471ab-9c53-48cd-9de4-9bd3918169de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:39.693441 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.693419 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8471ab-9c53-48cd-9de4-9bd3918169de-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:18:39.695164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.695134 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8471ab-9c53-48cd-9de4-9bd3918169de-kube-api-access-cc24v" pod "2d8471ab-9c53-48cd-9de4-9bd3918169de" (UID: "2d8471ab-9c53-48cd-9de4-9bd3918169de"). InnerVolumeSpecName "kube-api-access-cc24v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:39.695275 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.695152 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-maas-api-tls" pod "2d8471ab-9c53-48cd-9de4-9bd3918169de" (UID: "2d8471ab-9c53-48cd-9de4-9bd3918169de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:39.695704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.695682 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-metrics-certs" pod "2d8471ab-9c53-48cd-9de4-9bd3918169de" (UID: "2d8471ab-9c53-48cd-9de4-9bd3918169de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:39.794937 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.794897 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc24v\" (UniqueName: \"kubernetes.io/projected/2d8471ab-9c53-48cd-9de4-9bd3918169de-kube-api-access-cc24v\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:18:39.794937 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.794937 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:18:39.795169 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:39.794951 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8471ab-9c53-48cd-9de4-9bd3918169de-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:18:40.016942 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.016907 4625 generic.go:356] "Generic (PLEG): container finished" podID="2d8471ab-9c53-48cd-9de4-9bd3918169de" containerID="e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094" exitCode=0 Nov 03 23:18:40.017155 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.016986 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" Nov 03 23:18:40.017155 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.016993 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" event={"ID":"2d8471ab-9c53-48cd-9de4-9bd3918169de","Type":"ContainerDied","Data":"e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094"} Nov 03 23:18:40.017155 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.017059 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq" event={"ID":"2d8471ab-9c53-48cd-9de4-9bd3918169de","Type":"ContainerDied","Data":"68e0e5ff323fc64094e170e35bccaf33ef1269df65e8b84d2dbd90abd5e9b299"} Nov 03 23:18:40.017155 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.017088 4625 scope.go:122] "RemoveContainer" containerID="e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094" Nov 03 23:18:40.027340 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.027316 4625 scope.go:122] "RemoveContainer" containerID="e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094" Nov 03 23:18:40.027657 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:18:40.027632 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094\": container with ID starting with e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094 not found: ID does not exist" containerID="e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094" Nov 03 23:18:40.027710 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.027669 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094"} err="failed to get container status \"e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094\": rpc error: code = NotFound desc = could not find container \"e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094\": container with ID starting with e6294ec822fc821baf021751399055cfce5aac5f4a791c17b440afa445db2094 not found: ID does not exist" Nov 03 23:18:40.065793 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.065761 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq"] Nov 03 23:18:40.072594 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.072568 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-22c413-6bc44ff8c7-pqvsq"] Nov 03 23:18:40.741940 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:40.741905 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8471ab-9c53-48cd-9de4-9bd3918169de" path="/var/lib/kubelet/pods/2d8471ab-9c53-48cd-9de4-9bd3918169de/volumes" Nov 03 23:18:44.373343 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.373287 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" podUID="adc337f7-728b-44a2-8650-be5e763686ee" containerName="payload-pre-processing" containerID="cri-o://abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f" gracePeriod=25 Nov 03 23:18:44.648181 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.648146 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:18:44.743483 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.743431 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc337f7-728b-44a2-8650-be5e763686ee-kube-api-access-vth6n\" (UniqueName: \"kubernetes.io/projected/adc337f7-728b-44a2-8650-be5e763686ee-kube-api-access-vth6n\") pod \"adc337f7-728b-44a2-8650-be5e763686ee\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " Nov 03 23:18:44.743700 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.743557 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc337f7-728b-44a2-8650-be5e763686ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc337f7-728b-44a2-8650-be5e763686ee-plugins-config-volume\") pod \"adc337f7-728b-44a2-8650-be5e763686ee\" (UID: \"adc337f7-728b-44a2-8650-be5e763686ee\") " Nov 03 23:18:44.744000 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.743972 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc337f7-728b-44a2-8650-be5e763686ee-plugins-config-volume" pod "adc337f7-728b-44a2-8650-be5e763686ee" (UID: "adc337f7-728b-44a2-8650-be5e763686ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:44.745768 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.745712 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc337f7-728b-44a2-8650-be5e763686ee-kube-api-access-vth6n" pod "adc337f7-728b-44a2-8650-be5e763686ee" (UID: "adc337f7-728b-44a2-8650-be5e763686ee"). InnerVolumeSpecName "kube-api-access-vth6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:44.844985 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.844943 4625 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc337f7-728b-44a2-8650-be5e763686ee-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:18:44.844985 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:44.844975 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vth6n\" (UniqueName: \"kubernetes.io/projected/adc337f7-728b-44a2-8650-be5e763686ee-kube-api-access-vth6n\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:18:45.051855 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.051746 4625 generic.go:356] "Generic (PLEG): container finished" podID="adc337f7-728b-44a2-8650-be5e763686ee" containerID="abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f" exitCode=0 Nov 03 23:18:45.051855 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.051796 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" Nov 03 23:18:45.051855 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.051829 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" event={"ID":"adc337f7-728b-44a2-8650-be5e763686ee","Type":"ContainerDied","Data":"abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f"} Nov 03 23:18:45.052205 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.051896 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" event={"ID":"adc337f7-728b-44a2-8650-be5e763686ee","Type":"ContainerDied","Data":"0e5cdc51535fdfeb2b4e9a1c272a1a9774da616f535eff46050d7d0d6acc6d87"} Nov 03 23:18:45.052205 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.051925 4625 scope.go:122] "RemoveContainer" containerID="abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f" Nov 03 23:18:45.061533 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.061508 4625 scope.go:122] "RemoveContainer" containerID="abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f" Nov 03 23:18:45.061822 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:18:45.061804 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f\": container with ID starting with abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f not found: ID does not exist" containerID="abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f" Nov 03 23:18:45.061877 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.061830 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f"} err="failed to get container status \"abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f\": rpc error: code = NotFound desc = could not find container \"abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f\": container with ID starting with abd952d6d925a4257ba27abe053ba586b1ef3e3d49adcdee7d9d52d44ea2a37f not found: ID does not exist" Nov 03 23:18:45.101332 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.101291 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578"] Nov 03 23:18:45.112945 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.112893 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578"] Nov 03 23:18:45.609801 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:45.609759 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-22c413-748b6tg578" podUID="adc337f7-728b-44a2-8650-be5e763686ee" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.234:50052\" within 1s: context deadline exceeded" Nov 03 23:18:45.610237 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:18:45.609894 4625 logging.go:55] [core] [Channel #4719 SubChannel #4720]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.234:50052", ServerName: "10.134.0.234:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.234:50052: operation was canceled" Nov 03 23:18:46.727453 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:18:46.727413 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc337f7-728b-44a2-8650-be5e763686ee" path="/var/lib/kubelet/pods/adc337f7-728b-44a2-8650-be5e763686ee/volumes" Nov 03 23:19:02.185596 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:19:02.185559 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:19:07.941088 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:07.941056 4625 scope.go:122] "RemoveContainer" containerID="720745f3838daff190c5568503bfcc5f2ab3c3911c57dc8e44dce68e7cd99584" Nov 03 23:19:07.950120 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:07.950097 4625 scope.go:122] "RemoveContainer" containerID="9a29602dee6f781880769125ab86fc02bc802cd2c0a951543f763598667a9ac3" Nov 03 23:19:07.958700 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:07.958676 4625 scope.go:122] "RemoveContainer" containerID="49c1161694e063fd5d6d603c5cd148dc33986766c64841a6e3f29e152f97ae77" Nov 03 23:19:30.617231 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.617186 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj"] Nov 03 23:19:30.617876 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.617855 4625 cpu_manager.go:414] "containerMap: removing container" podUID="2d8471ab-9c53-48cd-9de4-9bd3918169de" containerName="maas-api" Nov 03 23:19:30.617930 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.617879 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8471ab-9c53-48cd-9de4-9bd3918169de" containerName="maas-api" Nov 03 23:19:30.617930 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.617921 4625 cpu_manager.go:414] "containerMap: removing container" podUID="adc337f7-728b-44a2-8650-be5e763686ee" containerName="payload-pre-processing" Nov 03 23:19:30.617998 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.617928 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc337f7-728b-44a2-8650-be5e763686ee" containerName="payload-pre-processing" Nov 03 23:19:30.618062 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.618049 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc337f7-728b-44a2-8650-be5e763686ee" containerName="payload-pre-processing" Nov 03 23:19:30.618099 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.618075 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8471ab-9c53-48cd-9de4-9bd3918169de" containerName="maas-api" Nov 03 23:19:30.623133 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.623109 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.639776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.639747 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj"] Nov 03 23:19:30.647352 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.647324 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:19:30.647577 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.647329 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:19:30.752264 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.752229 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.752264 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.752268 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.752527 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.752295 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbzs\" (UniqueName: \"kubernetes.io/projected/6cd3eb25-06aa-4f84-a19f-946149460fdf-kube-api-access-dnbzs\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.853807 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.853768 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.853807 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.853811 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.854108 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.853938 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbzs\" (UniqueName: \"kubernetes.io/projected/6cd3eb25-06aa-4f84-a19f-946149460fdf-kube-api-access-dnbzs\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.854641 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.854609 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.856455 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.856428 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.876266 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.876211 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbzs\" (UniqueName: \"kubernetes.io/projected/6cd3eb25-06aa-4f84-a19f-946149460fdf-kube-api-access-dnbzs\") pod \"maas-api-revoke-sub-452b6d7b-8nstj\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:30.939856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:30.939829 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:31.243366 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:31.243339 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj"] Nov 03 23:19:31.246247 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:19:31.246219 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd3eb25_06aa_4f84_a19f_946149460fdf.slice/crio-ed51c9b976719280d7e822a94d4e90c8cd7f738c1afb9579a0c7055c8a744371 WatchSource:0}: Error finding container ed51c9b976719280d7e822a94d4e90c8cd7f738c1afb9579a0c7055c8a744371: Status 404 returned error can't find the container with id ed51c9b976719280d7e822a94d4e90c8cd7f738c1afb9579a0c7055c8a744371 Nov 03 23:19:31.367420 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:31.367384 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" event={"ID":"6cd3eb25-06aa-4f84-a19f-946149460fdf","Type":"ContainerStarted","Data":"ed51c9b976719280d7e822a94d4e90c8cd7f738c1afb9579a0c7055c8a744371"} Nov 03 23:19:32.184434 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:19:32.184401 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:19:32.374507 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:32.374473 4625 generic.go:356] "Generic (PLEG): container finished" podID="6cd3eb25-06aa-4f84-a19f-946149460fdf" containerID="647549eb53b0f5dd6fa9aa9266cf497660f655c333e583de87d6d6b7f2fd0032" exitCode=0 Nov 03 23:19:32.374696 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:32.374546 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" event={"ID":"6cd3eb25-06aa-4f84-a19f-946149460fdf","Type":"ContainerDied","Data":"647549eb53b0f5dd6fa9aa9266cf497660f655c333e583de87d6d6b7f2fd0032"} Nov 03 23:19:33.561792 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.559423 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:33.684188 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.684090 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-openshift-ca\") pod \"6cd3eb25-06aa-4f84-a19f-946149460fdf\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " Nov 03 23:19:33.685572 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.685546 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd3eb25-06aa-4f84-a19f-946149460fdf-kube-api-access-dnbzs\" (UniqueName: \"kubernetes.io/projected/6cd3eb25-06aa-4f84-a19f-946149460fdf-kube-api-access-dnbzs\") pod \"6cd3eb25-06aa-4f84-a19f-946149460fdf\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " Nov 03 23:19:33.686210 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.686189 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-xks-ca\") pod \"6cd3eb25-06aa-4f84-a19f-946149460fdf\" (UID: \"6cd3eb25-06aa-4f84-a19f-946149460fdf\") " Nov 03 23:19:33.688304 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.685445 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-openshift-ca" pod "6cd3eb25-06aa-4f84-a19f-946149460fdf" (UID: "6cd3eb25-06aa-4f84-a19f-946149460fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:19:33.707975 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.706209 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-xks-ca" pod "6cd3eb25-06aa-4f84-a19f-946149460fdf" (UID: "6cd3eb25-06aa-4f84-a19f-946149460fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:19:33.708642 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.708617 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd3eb25-06aa-4f84-a19f-946149460fdf-kube-api-access-dnbzs" pod "6cd3eb25-06aa-4f84-a19f-946149460fdf" (UID: "6cd3eb25-06aa-4f84-a19f-946149460fdf"). InnerVolumeSpecName "kube-api-access-dnbzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:19:33.795329 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.795290 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnbzs\" (UniqueName: \"kubernetes.io/projected/6cd3eb25-06aa-4f84-a19f-946149460fdf-kube-api-access-dnbzs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:19:33.795582 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.795567 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:19:33.795685 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:33.795674 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3eb25-06aa-4f84-a19f-946149460fdf-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:19:34.406464 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:34.406423 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" event={"ID":"6cd3eb25-06aa-4f84-a19f-946149460fdf","Type":"ContainerDied","Data":"ed51c9b976719280d7e822a94d4e90c8cd7f738c1afb9579a0c7055c8a744371"} Nov 03 23:19:34.406748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:34.406705 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed51c9b976719280d7e822a94d4e90c8cd7f738c1afb9579a0c7055c8a744371" Nov 03 23:19:34.407034 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:34.406899 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj" Nov 03 23:19:35.290924 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:35.290871 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:19:35.291669 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:35.291612 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:19:46.835621 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.835531 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7"] Nov 03 23:19:46.836152 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.836100 4625 cpu_manager.go:414] "containerMap: removing container" podUID="6cd3eb25-06aa-4f84-a19f-946149460fdf" containerName="revoke-keys" Nov 03 23:19:46.836152 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.836115 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd3eb25-06aa-4f84-a19f-946149460fdf" containerName="revoke-keys" Nov 03 23:19:46.836305 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.836184 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd3eb25-06aa-4f84-a19f-946149460fdf" containerName="revoke-keys" Nov 03 23:19:46.839433 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.839408 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:46.867200 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.867173 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:19:46.867361 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.867349 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:19:46.870155 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.870133 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7"] Nov 03 23:19:46.937214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.937174 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f201a577-82e2-4218-8877-593ce43622d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:46.937385 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.937309 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f201a577-82e2-4218-8877-593ce43622d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:46.937385 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:46.937374 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpn5\" (UniqueName: \"kubernetes.io/projected/f201a577-82e2-4218-8877-593ce43622d1-kube-api-access-4vpn5\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.038761 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.038709 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f201a577-82e2-4218-8877-593ce43622d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.038939 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.038866 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f201a577-82e2-4218-8877-593ce43622d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.038939 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.038920 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpn5\" (UniqueName: \"kubernetes.io/projected/f201a577-82e2-4218-8877-593ce43622d1-kube-api-access-4vpn5\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.039365 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.039336 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f201a577-82e2-4218-8877-593ce43622d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.041270 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.041250 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f201a577-82e2-4218-8877-593ce43622d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.061040 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.061011 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpn5\" (UniqueName: \"kubernetes.io/projected/f201a577-82e2-4218-8877-593ce43622d1-kube-api-access-4vpn5\") pod \"maas-api-revoke-sub-3e31a506-plmv7\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.152524 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.152493 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:47.455096 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.455031 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7"] Nov 03 23:19:47.457482 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:19:47.457453 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf201a577_82e2_4218_8877_593ce43622d1.slice/crio-2ff45c4bc59d955d816a1a52392ea5813ed4425bd5222ebbc094ff11f6321a9f WatchSource:0}: Error finding container 2ff45c4bc59d955d816a1a52392ea5813ed4425bd5222ebbc094ff11f6321a9f: Status 404 returned error can't find the container with id 2ff45c4bc59d955d816a1a52392ea5813ed4425bd5222ebbc094ff11f6321a9f Nov 03 23:19:47.532392 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:47.532355 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" event={"ID":"f201a577-82e2-4218-8877-593ce43622d1","Type":"ContainerStarted","Data":"2ff45c4bc59d955d816a1a52392ea5813ed4425bd5222ebbc094ff11f6321a9f"} Nov 03 23:19:48.542441 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:48.542354 4625 generic.go:356] "Generic (PLEG): container finished" podID="f201a577-82e2-4218-8877-593ce43622d1" containerID="c69d0a6d4f35a2324038787abe03f5a92b1352072cde0174ae3faae3abf00436" exitCode=0 Nov 03 23:19:48.543017 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:48.542395 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" event={"ID":"f201a577-82e2-4218-8877-593ce43622d1","Type":"ContainerDied","Data":"c69d0a6d4f35a2324038787abe03f5a92b1352072cde0174ae3faae3abf00436"} Nov 03 23:19:50.030178 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.030153 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:50.122771 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.122737 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f201a577-82e2-4218-8877-593ce43622d1-kube-api-access-4vpn5\" (UniqueName: \"kubernetes.io/projected/f201a577-82e2-4218-8877-593ce43622d1-kube-api-access-4vpn5\") pod \"f201a577-82e2-4218-8877-593ce43622d1\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " Nov 03 23:19:50.122944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.122783 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f201a577-82e2-4218-8877-593ce43622d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f201a577-82e2-4218-8877-593ce43622d1-maas-api-xks-ca\") pod \"f201a577-82e2-4218-8877-593ce43622d1\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " Nov 03 23:19:50.122944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.122829 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f201a577-82e2-4218-8877-593ce43622d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f201a577-82e2-4218-8877-593ce43622d1-maas-api-openshift-ca\") pod \"f201a577-82e2-4218-8877-593ce43622d1\" (UID: \"f201a577-82e2-4218-8877-593ce43622d1\") " Nov 03 23:19:50.123366 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.123342 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f201a577-82e2-4218-8877-593ce43622d1-maas-api-openshift-ca" pod "f201a577-82e2-4218-8877-593ce43622d1" (UID: "f201a577-82e2-4218-8877-593ce43622d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:19:50.124946 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.124879 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f201a577-82e2-4218-8877-593ce43622d1-kube-api-access-4vpn5" pod "f201a577-82e2-4218-8877-593ce43622d1" (UID: "f201a577-82e2-4218-8877-593ce43622d1"). InnerVolumeSpecName "kube-api-access-4vpn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:19:50.125515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.125492 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f201a577-82e2-4218-8877-593ce43622d1-maas-api-xks-ca" pod "f201a577-82e2-4218-8877-593ce43622d1" (UID: "f201a577-82e2-4218-8877-593ce43622d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:19:50.225150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.225118 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpn5\" (UniqueName: \"kubernetes.io/projected/f201a577-82e2-4218-8877-593ce43622d1-kube-api-access-4vpn5\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:19:50.225150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.225146 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f201a577-82e2-4218-8877-593ce43622d1-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:19:50.225150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.225156 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f201a577-82e2-4218-8877-593ce43622d1-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:19:50.616318 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.616279 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" event={"ID":"f201a577-82e2-4218-8877-593ce43622d1","Type":"ContainerDied","Data":"2ff45c4bc59d955d816a1a52392ea5813ed4425bd5222ebbc094ff11f6321a9f"} Nov 03 23:19:50.616640 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.616618 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff45c4bc59d955d816a1a52392ea5813ed4425bd5222ebbc094ff11f6321a9f" Nov 03 23:19:50.616861 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.616843 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7" Nov 03 23:19:50.816893 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.816832 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gprr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:19:50.817127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:19:50.816909 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gprr2" podUID="e6eb71bf-3a09-4a93-b518-91961518a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:20:02.184201 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:20:02.184160 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:20:05.297357 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:05.296823 4625 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-mfkdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:20:05.297357 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:05.296907 4625 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-mfkdg" podUID="13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:20:07.997041 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:07.997008 4625 scope.go:122] "RemoveContainer" containerID="f4775c5b2c3dc9781a4826f3fa34f21f4054e118d5a813772e985a53d011030a" Nov 03 23:20:09.314050 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.314018 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh"] Nov 03 23:20:09.314444 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.314434 4625 cpu_manager.go:414] "containerMap: removing container" podUID="f201a577-82e2-4218-8877-593ce43622d1" containerName="revoke-keys" Nov 03 23:20:09.314497 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.314445 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f201a577-82e2-4218-8877-593ce43622d1" containerName="revoke-keys" Nov 03 23:20:09.314536 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.314525 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="f201a577-82e2-4218-8877-593ce43622d1" containerName="revoke-keys" Nov 03 23:20:09.317711 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.317692 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.337701 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.337673 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh"] Nov 03 23:20:09.345306 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.345286 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:20:09.345445 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.345333 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:20:09.359900 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.359874 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.360021 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.360006 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgn4s\" (UniqueName: \"kubernetes.io/projected/e29c0007-a41c-4906-b2cf-cfd8f9259991-kube-api-access-lgn4s\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.360075 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.360044 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.461150 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.461116 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn4s\" (UniqueName: \"kubernetes.io/projected/e29c0007-a41c-4906-b2cf-cfd8f9259991-kube-api-access-lgn4s\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.461356 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.461185 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.461356 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.461283 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.462063 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.462038 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.463759 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.463736 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.483177 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.483151 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn4s\" (UniqueName: \"kubernetes.io/projected/e29c0007-a41c-4906-b2cf-cfd8f9259991-kube-api-access-lgn4s\") pod \"maas-api-revoke-sub-b811c3ac-996zh\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.630958 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.630922 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:09.934683 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:09.934657 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh"] Nov 03 23:20:09.937656 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:20:09.937615 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29c0007_a41c_4906_b2cf_cfd8f9259991.slice/crio-d4541599bad98d54564cc7cd38f4d3f751295832b6218cd0b873d46af7d02b74 WatchSource:0}: Error finding container d4541599bad98d54564cc7cd38f4d3f751295832b6218cd0b873d46af7d02b74: Status 404 returned error can't find the container with id d4541599bad98d54564cc7cd38f4d3f751295832b6218cd0b873d46af7d02b74 Nov 03 23:20:10.747690 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:10.747664 4625 generic.go:356] "Generic (PLEG): container finished" podID="e29c0007-a41c-4906-b2cf-cfd8f9259991" containerID="e0ce8b6fd7afe29012cca0271556d1902f88b7296fa327b80dd33ef2c5988a22" exitCode=0 Nov 03 23:20:10.748041 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:10.747748 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" event={"ID":"e29c0007-a41c-4906-b2cf-cfd8f9259991","Type":"ContainerDied","Data":"e0ce8b6fd7afe29012cca0271556d1902f88b7296fa327b80dd33ef2c5988a22"} Nov 03 23:20:10.748041 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:10.747817 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" event={"ID":"e29c0007-a41c-4906-b2cf-cfd8f9259991","Type":"ContainerStarted","Data":"d4541599bad98d54564cc7cd38f4d3f751295832b6218cd0b873d46af7d02b74"} Nov 03 23:20:11.894348 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:11.894326 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:11.983837 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:11.983806 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29c0007-a41c-4906-b2cf-cfd8f9259991-kube-api-access-lgn4s\" (UniqueName: \"kubernetes.io/projected/e29c0007-a41c-4906-b2cf-cfd8f9259991-kube-api-access-lgn4s\") pod \"e29c0007-a41c-4906-b2cf-cfd8f9259991\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " Nov 03 23:20:11.984013 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:11.983845 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-openshift-ca\") pod \"e29c0007-a41c-4906-b2cf-cfd8f9259991\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " Nov 03 23:20:11.984013 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:11.983881 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-xks-ca\") pod \"e29c0007-a41c-4906-b2cf-cfd8f9259991\" (UID: \"e29c0007-a41c-4906-b2cf-cfd8f9259991\") " Nov 03 23:20:11.984301 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:11.984264 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-openshift-ca" pod "e29c0007-a41c-4906-b2cf-cfd8f9259991" (UID: "e29c0007-a41c-4906-b2cf-cfd8f9259991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:20:11.985987 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:11.985960 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-xks-ca" pod "e29c0007-a41c-4906-b2cf-cfd8f9259991" (UID: "e29c0007-a41c-4906-b2cf-cfd8f9259991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:20:11.986117 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:11.985983 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29c0007-a41c-4906-b2cf-cfd8f9259991-kube-api-access-lgn4s" pod "e29c0007-a41c-4906-b2cf-cfd8f9259991" (UID: "e29c0007-a41c-4906-b2cf-cfd8f9259991"). InnerVolumeSpecName "kube-api-access-lgn4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:20:12.085236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:12.085167 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-xks-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:20:12.085236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:12.085195 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgn4s\" (UniqueName: \"kubernetes.io/projected/e29c0007-a41c-4906-b2cf-cfd8f9259991-kube-api-access-lgn4s\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:20:12.085236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:12.085205 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29c0007-a41c-4906-b2cf-cfd8f9259991-maas-api-openshift-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:20:12.760404 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:12.760372 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" Nov 03 23:20:12.760577 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:12.760366 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh" event={"ID":"e29c0007-a41c-4906-b2cf-cfd8f9259991","Type":"ContainerDied","Data":"d4541599bad98d54564cc7cd38f4d3f751295832b6218cd0b873d46af7d02b74"} Nov 03 23:20:12.760577 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:12.760497 4625 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4541599bad98d54564cc7cd38f4d3f751295832b6218cd0b873d46af7d02b74" Nov 03 23:20:21.698035 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.697995 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n"] Nov 03 23:20:21.698659 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.698643 4625 cpu_manager.go:414] "containerMap: removing container" podUID="e29c0007-a41c-4906-b2cf-cfd8f9259991" containerName="revoke-keys" Nov 03 23:20:21.698748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.698661 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29c0007-a41c-4906-b2cf-cfd8f9259991" containerName="revoke-keys" Nov 03 23:20:21.698803 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.698773 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29c0007-a41c-4906-b2cf-cfd8f9259991" containerName="revoke-keys" Nov 03 23:20:21.702237 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.702208 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.714420 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.714391 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds"] Nov 03 23:20:21.718619 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.718593 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.726745 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.726687 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n"] Nov 03 23:20:21.731500 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.731474 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qn4p2\"" Nov 03 23:20:21.731630 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.731555 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 23:20:21.731630 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.731603 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:20:21.743229 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.743194 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds"] Nov 03 23:20:21.762190 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762160 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b2b60f-fe33-47e0-a34e-57e004232598-openshift-service-ca\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.762380 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762203 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-maas-api-tls\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.762380 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762222 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8535d0ca-b280-4d43-abc1-2d9643eef8e8-maas-api-tls\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.762488 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762463 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gft\" (UniqueName: \"kubernetes.io/projected/8535d0ca-b280-4d43-abc1-2d9643eef8e8-kube-api-access-d7gft\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.762560 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762539 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-metrics-certs\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.762626 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762587 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d0ca-b280-4d43-abc1-2d9643eef8e8-openshift-service-ca\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.762694 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762624 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghkm\" (UniqueName: \"kubernetes.io/projected/01b2b60f-fe33-47e0-a34e-57e004232598-kube-api-access-bghkm\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.762832 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.762807 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8535d0ca-b280-4d43-abc1-2d9643eef8e8-metrics-certs\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.863767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863701 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b2b60f-fe33-47e0-a34e-57e004232598-openshift-service-ca\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.863980 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863779 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-maas-api-tls\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.863980 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863806 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8535d0ca-b280-4d43-abc1-2d9643eef8e8-maas-api-tls\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.863980 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863848 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gft\" (UniqueName: \"kubernetes.io/projected/8535d0ca-b280-4d43-abc1-2d9643eef8e8-kube-api-access-d7gft\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.863980 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863896 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-metrics-certs\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.863980 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863925 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d0ca-b280-4d43-abc1-2d9643eef8e8-openshift-service-ca\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.863980 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863950 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghkm\" (UniqueName: \"kubernetes.io/projected/01b2b60f-fe33-47e0-a34e-57e004232598-kube-api-access-bghkm\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.864324 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.863998 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8535d0ca-b280-4d43-abc1-2d9643eef8e8-metrics-certs\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.864646 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.864612 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b2b60f-fe33-47e0-a34e-57e004232598-openshift-service-ca\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.864776 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.864668 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d0ca-b280-4d43-abc1-2d9643eef8e8-openshift-service-ca\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.866560 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.866534 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8535d0ca-b280-4d43-abc1-2d9643eef8e8-maas-api-tls\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.866662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.866592 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8535d0ca-b280-4d43-abc1-2d9643eef8e8-metrics-certs\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:21.866662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.866615 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-maas-api-tls\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.867519 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.867497 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-metrics-certs\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.886280 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.886251 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghkm\" (UniqueName: \"kubernetes.io/projected/01b2b60f-fe33-47e0-a34e-57e004232598-kube-api-access-bghkm\") pod \"maas-api-84f7fbd778-9dl9n\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:21.886472 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:21.886438 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gft\" (UniqueName: \"kubernetes.io/projected/8535d0ca-b280-4d43-abc1-2d9643eef8e8-kube-api-access-d7gft\") pod \"maas-api-5d897ffcf4-7fnds\" (UID: \"8535d0ca-b280-4d43-abc1-2d9643eef8e8\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:22.021488 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.021409 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:22.042211 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.042180 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:22.334631 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.334596 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n"] Nov 03 23:20:22.338795 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.338772 4625 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:20:22.376675 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.376638 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds"] Nov 03 23:20:22.378608 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:20:22.378581 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8535d0ca_b280_4d43_abc1_2d9643eef8e8.slice/crio-8621fa947ad57f1a4045e1c12bd9dc32b64b032ee33b71e8a781890488ae13df WatchSource:0}: Error finding container 8621fa947ad57f1a4045e1c12bd9dc32b64b032ee33b71e8a781890488ae13df: Status 404 returned error can't find the container with id 8621fa947ad57f1a4045e1c12bd9dc32b64b032ee33b71e8a781890488ae13df Nov 03 23:20:22.823179 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.823143 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" event={"ID":"01b2b60f-fe33-47e0-a34e-57e004232598","Type":"ContainerStarted","Data":"cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79"} Nov 03 23:20:22.823672 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.823213 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" event={"ID":"01b2b60f-fe33-47e0-a34e-57e004232598","Type":"ContainerStarted","Data":"ffe017d7cfdd95b01f54959d6e9d04b6580175b0f547eb698e25a81541354fe6"} Nov 03 23:20:22.823672 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.823350 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:22.825033 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.825009 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" event={"ID":"8535d0ca-b280-4d43-abc1-2d9643eef8e8","Type":"ContainerStarted","Data":"bc66f8e312254401c131eb7646469c88c56fe78fd18ec2e9db6925e99c88a72a"} Nov 03 23:20:22.825161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.825052 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" event={"ID":"8535d0ca-b280-4d43-abc1-2d9643eef8e8","Type":"ContainerStarted","Data":"8621fa947ad57f1a4045e1c12bd9dc32b64b032ee33b71e8a781890488ae13df"} Nov 03 23:20:22.825161 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.825151 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:22.880216 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.880155 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" podStartSLOduration=1.5063582869999999 podStartE2EDuration="1.880137029s" totalImagesPullingTime="373.778742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:22.878571164 +0000 UTC m=+4044.307907157" watchObservedRunningTime="2026-11-03 23:20:22.880137029 +0000 UTC m=+4044.309473027" Nov 03 23:20:22.920528 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:22.920471 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" podStartSLOduration=1.561704076 podStartE2EDuration="1.92045684s" totalImagesPullingTime="358.752764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:22.91728061 +0000 UTC m=+4044.346616603" watchObservedRunningTime="2026-11-03 23:20:22.92045684 +0000 UTC m=+4044.349792833" Nov 03 23:20:28.837236 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:28.837204 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:28.837767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:28.837391 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-7fnds" Nov 03 23:20:29.030398 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.030351 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n"] Nov 03 23:20:29.030708 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.030662 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" podUID="01b2b60f-fe33-47e0-a34e-57e004232598" containerName="maas-api" containerID="cri-o://cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79" gracePeriod=30 Nov 03 23:20:29.305929 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.305899 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:29.345976 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.345940 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b2b60f-fe33-47e0-a34e-57e004232598-kube-api-access-bghkm\" (UniqueName: \"kubernetes.io/projected/01b2b60f-fe33-47e0-a34e-57e004232598-kube-api-access-bghkm\") pod \"01b2b60f-fe33-47e0-a34e-57e004232598\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " Nov 03 23:20:29.346225 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.345990 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-metrics-certs\") pod \"01b2b60f-fe33-47e0-a34e-57e004232598\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " Nov 03 23:20:29.346225 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.346038 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b2b60f-fe33-47e0-a34e-57e004232598-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b2b60f-fe33-47e0-a34e-57e004232598-openshift-service-ca\") pod \"01b2b60f-fe33-47e0-a34e-57e004232598\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " Nov 03 23:20:29.346225 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.346134 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-maas-api-tls\") pod \"01b2b60f-fe33-47e0-a34e-57e004232598\" (UID: \"01b2b60f-fe33-47e0-a34e-57e004232598\") " Nov 03 23:20:29.347895 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.346700 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b2b60f-fe33-47e0-a34e-57e004232598-openshift-service-ca" pod "01b2b60f-fe33-47e0-a34e-57e004232598" (UID: "01b2b60f-fe33-47e0-a34e-57e004232598"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:20:29.349866 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.349825 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-metrics-certs" pod "01b2b60f-fe33-47e0-a34e-57e004232598" (UID: "01b2b60f-fe33-47e0-a34e-57e004232598"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:20:29.350090 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.350068 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-maas-api-tls" pod "01b2b60f-fe33-47e0-a34e-57e004232598" (UID: "01b2b60f-fe33-47e0-a34e-57e004232598"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:20:29.350181 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.350166 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b2b60f-fe33-47e0-a34e-57e004232598-kube-api-access-bghkm" pod "01b2b60f-fe33-47e0-a34e-57e004232598" (UID: "01b2b60f-fe33-47e0-a34e-57e004232598"). InnerVolumeSpecName "kube-api-access-bghkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:20:29.447394 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.447302 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b2b60f-fe33-47e0-a34e-57e004232598-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:20:29.447394 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.447337 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:20:29.447394 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.447346 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghkm\" (UniqueName: \"kubernetes.io/projected/01b2b60f-fe33-47e0-a34e-57e004232598-kube-api-access-bghkm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:20:29.447394 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.447355 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b2b60f-fe33-47e0-a34e-57e004232598-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:20:29.869994 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.869963 4625 generic.go:356] "Generic (PLEG): container finished" podID="01b2b60f-fe33-47e0-a34e-57e004232598" containerID="cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79" exitCode=0 Nov 03 23:20:29.870515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.870024 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" event={"ID":"01b2b60f-fe33-47e0-a34e-57e004232598","Type":"ContainerDied","Data":"cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79"} Nov 03 23:20:29.870515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.870029 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" Nov 03 23:20:29.870515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.870065 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n" event={"ID":"01b2b60f-fe33-47e0-a34e-57e004232598","Type":"ContainerDied","Data":"ffe017d7cfdd95b01f54959d6e9d04b6580175b0f547eb698e25a81541354fe6"} Nov 03 23:20:29.870515 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.870087 4625 scope.go:122] "RemoveContainer" containerID="cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79" Nov 03 23:20:29.881860 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.881829 4625 scope.go:122] "RemoveContainer" containerID="cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79" Nov 03 23:20:29.882149 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:20:29.882128 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79\": container with ID starting with cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79 not found: ID does not exist" containerID="cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79" Nov 03 23:20:29.882219 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.882156 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79"} err="failed to get container status \"cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79\": rpc error: code = NotFound desc = could not find container \"cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79\": container with ID starting with cac8e76b26ea81e390eda1db224ff6f5531f24cac51c11818e600a4904150c79 not found: ID does not exist" Nov 03 23:20:29.921411 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.921375 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n"] Nov 03 23:20:29.932435 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:29.932394 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-9dl9n"] Nov 03 23:20:30.727286 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:30.727254 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b2b60f-fe33-47e0-a34e-57e004232598" path="/var/lib/kubelet/pods/01b2b60f-fe33-47e0-a34e-57e004232598/volumes" Nov 03 23:20:32.186613 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:20:32.186575 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:20:33.458352 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.458316 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7"] Nov 03 23:20:33.458869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.458829 4625 cpu_manager.go:414] "containerMap: removing container" podUID="01b2b60f-fe33-47e0-a34e-57e004232598" containerName="maas-api" Nov 03 23:20:33.458869 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.458844 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b2b60f-fe33-47e0-a34e-57e004232598" containerName="maas-api" Nov 03 23:20:33.458951 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.458919 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b2b60f-fe33-47e0-a34e-57e004232598" containerName="maas-api" Nov 03 23:20:33.463616 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.463587 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.490041 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.489995 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a6ecd9bf-serving-cert\"" Nov 03 23:20:33.493272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.493243 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7"] Nov 03 23:20:33.595673 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.595628 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvldb\" (UniqueName: \"kubernetes.io/projected/3711ecdb-22b1-4bcb-81fa-8bc691459c52-kube-api-access-xvldb\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.595882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.595684 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3711ecdb-22b1-4bcb-81fa-8bc691459c52-openshift-service-ca\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.595882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.595792 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-metrics-certs\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.595882 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.595866 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-maas-api-tls\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.697491 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.697456 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvldb\" (UniqueName: \"kubernetes.io/projected/3711ecdb-22b1-4bcb-81fa-8bc691459c52-kube-api-access-xvldb\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.697491 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.697499 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3711ecdb-22b1-4bcb-81fa-8bc691459c52-openshift-service-ca\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.697794 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.697538 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-metrics-certs\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.697794 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.697644 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-maas-api-tls\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.698214 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.698191 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3711ecdb-22b1-4bcb-81fa-8bc691459c52-openshift-service-ca\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.700209 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.700182 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-maas-api-tls\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.700306 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.700221 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-metrics-certs\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.720554 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.720479 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvldb\" (UniqueName: \"kubernetes.io/projected/3711ecdb-22b1-4bcb-81fa-8bc691459c52-kube-api-access-xvldb\") pod \"maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:33.777587 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:33.777558 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:34.087284 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:34.087206 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7"] Nov 03 23:20:34.092161 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:20:34.092129 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3711ecdb_22b1_4bcb_81fa_8bc691459c52.slice/crio-b7bcf1fed61e5245a9163a60bed0ae1d4f17e84f9121b5f3aa73c08114fbb5f2 WatchSource:0}: Error finding container b7bcf1fed61e5245a9163a60bed0ae1d4f17e84f9121b5f3aa73c08114fbb5f2: Status 404 returned error can't find the container with id b7bcf1fed61e5245a9163a60bed0ae1d4f17e84f9121b5f3aa73c08114fbb5f2 Nov 03 23:20:34.907879 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:34.907841 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" event={"ID":"3711ecdb-22b1-4bcb-81fa-8bc691459c52","Type":"ContainerStarted","Data":"00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388"} Nov 03 23:20:34.908297 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:34.907891 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" event={"ID":"3711ecdb-22b1-4bcb-81fa-8bc691459c52","Type":"ContainerStarted","Data":"b7bcf1fed61e5245a9163a60bed0ae1d4f17e84f9121b5f3aa73c08114fbb5f2"} Nov 03 23:20:34.908297 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:34.907924 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:20:34.950628 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:34.950571 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" podStartSLOduration=1.678342419 podStartE2EDuration="1.950555528s" totalImagesPullingTime="272.213109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:34.948366902 +0000 UTC m=+4056.377702896" watchObservedRunningTime="2026-11-03 23:20:34.950555528 +0000 UTC m=+4056.379891520" Nov 03 23:20:40.918893 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:20:40.918860 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:21:00.106142 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.106099 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7"] Nov 03 23:21:00.106898 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.106640 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" podUID="3711ecdb-22b1-4bcb-81fa-8bc691459c52" containerName="maas-api" containerID="cri-o://00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388" gracePeriod=30 Nov 03 23:21:00.386676 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.386651 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:21:00.484670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.484632 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-metrics-certs\") pod \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " Nov 03 23:21:00.484874 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.484796 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3711ecdb-22b1-4bcb-81fa-8bc691459c52-kube-api-access-xvldb\" (UniqueName: \"kubernetes.io/projected/3711ecdb-22b1-4bcb-81fa-8bc691459c52-kube-api-access-xvldb\") pod \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " Nov 03 23:21:00.484874 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.484862 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3711ecdb-22b1-4bcb-81fa-8bc691459c52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3711ecdb-22b1-4bcb-81fa-8bc691459c52-openshift-service-ca\") pod \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " Nov 03 23:21:00.485003 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.484927 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-maas-api-tls\") pod \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\" (UID: \"3711ecdb-22b1-4bcb-81fa-8bc691459c52\") " Nov 03 23:21:00.485363 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.485287 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3711ecdb-22b1-4bcb-81fa-8bc691459c52-openshift-service-ca" pod "3711ecdb-22b1-4bcb-81fa-8bc691459c52" (UID: "3711ecdb-22b1-4bcb-81fa-8bc691459c52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:21:00.487453 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.487422 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-metrics-certs" pod "3711ecdb-22b1-4bcb-81fa-8bc691459c52" (UID: "3711ecdb-22b1-4bcb-81fa-8bc691459c52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:21:00.487598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.487443 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-maas-api-tls" pod "3711ecdb-22b1-4bcb-81fa-8bc691459c52" (UID: "3711ecdb-22b1-4bcb-81fa-8bc691459c52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:21:00.487598 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.487461 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3711ecdb-22b1-4bcb-81fa-8bc691459c52-kube-api-access-xvldb" pod "3711ecdb-22b1-4bcb-81fa-8bc691459c52" (UID: "3711ecdb-22b1-4bcb-81fa-8bc691459c52"). InnerVolumeSpecName "kube-api-access-xvldb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:21:00.589808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.587016 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:21:00.589808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.587050 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3711ecdb-22b1-4bcb-81fa-8bc691459c52-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:21:00.589808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.587065 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvldb\" (UniqueName: \"kubernetes.io/projected/3711ecdb-22b1-4bcb-81fa-8bc691459c52-kube-api-access-xvldb\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:21:00.589808 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:00.587086 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3711ecdb-22b1-4bcb-81fa-8bc691459c52-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:21:01.080860 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.080776 4625 generic.go:356] "Generic (PLEG): container finished" podID="3711ecdb-22b1-4bcb-81fa-8bc691459c52" containerID="00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388" exitCode=0 Nov 03 23:21:01.080860 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.080829 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" Nov 03 23:21:01.080860 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.080845 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" event={"ID":"3711ecdb-22b1-4bcb-81fa-8bc691459c52","Type":"ContainerDied","Data":"00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388"} Nov 03 23:21:01.081127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.080908 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7" event={"ID":"3711ecdb-22b1-4bcb-81fa-8bc691459c52","Type":"ContainerDied","Data":"b7bcf1fed61e5245a9163a60bed0ae1d4f17e84f9121b5f3aa73c08114fbb5f2"} Nov 03 23:21:01.081127 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.080934 4625 scope.go:122] "RemoveContainer" containerID="00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388" Nov 03 23:21:01.090668 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.090649 4625 scope.go:122] "RemoveContainer" containerID="00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388" Nov 03 23:21:01.090953 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:21:01.090932 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388\": container with ID starting with 00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388 not found: ID does not exist" containerID="00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388" Nov 03 23:21:01.091007 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.090962 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388"} err="failed to get container status \"00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388\": rpc error: code = NotFound desc = could not find container \"00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388\": container with ID starting with 00c90fec334c4ca93e8f88a0cd39bf8707e948e87593431586736faf9e46d388 not found: ID does not exist" Nov 03 23:21:01.131488 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.131456 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7"] Nov 03 23:21:01.141707 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:01.141675 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6ecd9bf-bf444f9b5-nqcq7"] Nov 03 23:21:02.187098 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:21:02.187063 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:21:02.727802 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:02.727764 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3711ecdb-22b1-4bcb-81fa-8bc691459c52" path="/var/lib/kubelet/pods/3711ecdb-22b1-4bcb-81fa-8bc691459c52/volumes" Nov 03 23:21:10.252735 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.252678 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5"] Nov 03 23:21:10.253613 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.253581 4625 cpu_manager.go:414] "containerMap: removing container" podUID="3711ecdb-22b1-4bcb-81fa-8bc691459c52" containerName="maas-api" Nov 03 23:21:10.253767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.253615 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3711ecdb-22b1-4bcb-81fa-8bc691459c52" containerName="maas-api" Nov 03 23:21:10.253767 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.253740 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="3711ecdb-22b1-4bcb-81fa-8bc691459c52" containerName="maas-api" Nov 03 23:21:10.258271 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.258246 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.275978 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.275948 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-02e08d-serving-cert\"" Nov 03 23:21:10.290688 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.290648 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5"] Nov 03 23:21:10.294535 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.294498 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.294748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.294614 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a058ab5c-c053-4cca-aa99-f935d9292448-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.294748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.294650 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-metrics-certs\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.294748 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.294679 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbs2\" (UniqueName: \"kubernetes.io/projected/a058ab5c-c053-4cca-aa99-f935d9292448-kube-api-access-bjbs2\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.396084 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.396044 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.396291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.396124 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a058ab5c-c053-4cca-aa99-f935d9292448-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.396291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.396153 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-metrics-certs\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.396291 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.396173 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbs2\" (UniqueName: \"kubernetes.io/projected/a058ab5c-c053-4cca-aa99-f935d9292448-kube-api-access-bjbs2\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.397413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.397114 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a058ab5c-c053-4cca-aa99-f935d9292448-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.399175 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.399146 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-metrics-certs\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.399273 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.399187 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.418121 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.418089 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbs2\" (UniqueName: \"kubernetes.io/projected/a058ab5c-c053-4cca-aa99-f935d9292448-kube-api-access-bjbs2\") pod \"maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.571198 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.571106 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:10.926774 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:10.926711 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5"] Nov 03 23:21:10.929751 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:21:10.929726 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda058ab5c_c053_4cca_aa99_f935d9292448.slice/crio-ebd707fddff497e1cbb11b2e7a7e21e0bd43265fa8b7052b0ac8aea349e5495d WatchSource:0}: Error finding container ebd707fddff497e1cbb11b2e7a7e21e0bd43265fa8b7052b0ac8aea349e5495d: Status 404 returned error can't find the container with id ebd707fddff497e1cbb11b2e7a7e21e0bd43265fa8b7052b0ac8aea349e5495d Nov 03 23:21:11.142606 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:11.142568 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" event={"ID":"a058ab5c-c053-4cca-aa99-f935d9292448","Type":"ContainerStarted","Data":"ebd707fddff497e1cbb11b2e7a7e21e0bd43265fa8b7052b0ac8aea349e5495d"} Nov 03 23:21:12.152264 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:12.152228 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" event={"ID":"a058ab5c-c053-4cca-aa99-f935d9292448","Type":"ContainerStarted","Data":"df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7"} Nov 03 23:21:12.152686 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:12.152364 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:12.194450 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:12.194394 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" podStartSLOduration=1.860869327 podStartE2EDuration="2.194377075s" totalImagesPullingTime="333.507748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:12.191460704 +0000 UTC m=+4093.620796697" watchObservedRunningTime="2026-11-03 23:21:12.194377075 +0000 UTC m=+4093.623713069" Nov 03 23:21:18.165385 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:18.165352 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:21:32.186662 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:21:32.186624 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:21:57.031545 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:57.031502 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz"] Nov 03 23:21:57.042542 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:57.042516 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vv4hz"] Nov 03 23:21:58.726467 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:21:58.726435 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76477a57-ab89-4bc8-b993-044b447788b4" path="/var/lib/kubelet/pods/76477a57-ab89-4bc8-b993-044b447788b4/volumes" Nov 03 23:22:02.184819 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:22:02.184777 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:22:18.027466 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:22:18.027425 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh"] Nov 03 23:22:18.038157 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:22:18.038128 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cq9kh"] Nov 03 23:22:18.727054 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:22:18.727022 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a819ab71-48bd-45fd-ad42-ab4a9f9e85a7" path="/var/lib/kubelet/pods/a819ab71-48bd-45fd-ad42-ab4a9f9e85a7/volumes" Nov 03 23:22:32.186623 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:22:32.186583 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:23:01.407680 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.407635 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5"] Nov 03 23:23:01.408296 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.407979 4625 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" podUID="a058ab5c-c053-4cca-aa99-f935d9292448" containerName="maas-api" containerID="cri-o://df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7" gracePeriod=30 Nov 03 23:23:01.702130 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.702098 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:23:01.759225 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.759191 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-maas-api-tls\") pod \"a058ab5c-c053-4cca-aa99-f935d9292448\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " Nov 03 23:23:01.759413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.759240 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a058ab5c-c053-4cca-aa99-f935d9292448-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a058ab5c-c053-4cca-aa99-f935d9292448-openshift-service-ca\") pod \"a058ab5c-c053-4cca-aa99-f935d9292448\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " Nov 03 23:23:01.759413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.759259 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-metrics-certs\") pod \"a058ab5c-c053-4cca-aa99-f935d9292448\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " Nov 03 23:23:01.759413 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.759286 4625 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a058ab5c-c053-4cca-aa99-f935d9292448-kube-api-access-bjbs2\" (UniqueName: \"kubernetes.io/projected/a058ab5c-c053-4cca-aa99-f935d9292448-kube-api-access-bjbs2\") pod \"a058ab5c-c053-4cca-aa99-f935d9292448\" (UID: \"a058ab5c-c053-4cca-aa99-f935d9292448\") " Nov 03 23:23:01.759688 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.759657 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a058ab5c-c053-4cca-aa99-f935d9292448-openshift-service-ca" pod "a058ab5c-c053-4cca-aa99-f935d9292448" (UID: "a058ab5c-c053-4cca-aa99-f935d9292448"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:23:01.761565 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.761530 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-maas-api-tls" pod "a058ab5c-c053-4cca-aa99-f935d9292448" (UID: "a058ab5c-c053-4cca-aa99-f935d9292448"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:23:01.761565 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.761556 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-metrics-certs" pod "a058ab5c-c053-4cca-aa99-f935d9292448" (UID: "a058ab5c-c053-4cca-aa99-f935d9292448"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:23:01.761793 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.761611 4625 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a058ab5c-c053-4cca-aa99-f935d9292448-kube-api-access-bjbs2" pod "a058ab5c-c053-4cca-aa99-f935d9292448" (UID: "a058ab5c-c053-4cca-aa99-f935d9292448"). InnerVolumeSpecName "kube-api-access-bjbs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:01.833692 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.833659 4625 generic.go:356] "Generic (PLEG): container finished" podID="a058ab5c-c053-4cca-aa99-f935d9292448" containerID="df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7" exitCode=0 Nov 03 23:23:01.833888 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.833706 4625 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" Nov 03 23:23:01.833888 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.833754 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" event={"ID":"a058ab5c-c053-4cca-aa99-f935d9292448","Type":"ContainerDied","Data":"df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7"} Nov 03 23:23:01.833888 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.833814 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5" event={"ID":"a058ab5c-c053-4cca-aa99-f935d9292448","Type":"ContainerDied","Data":"ebd707fddff497e1cbb11b2e7a7e21e0bd43265fa8b7052b0ac8aea349e5495d"} Nov 03 23:23:01.833888 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.833844 4625 scope.go:122] "RemoveContainer" containerID="df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7" Nov 03 23:23:01.844302 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.844280 4625 scope.go:122] "RemoveContainer" containerID="df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7" Nov 03 23:23:01.844613 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:23:01.844586 4625 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7\": container with ID starting with df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7 not found: ID does not exist" containerID="df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7" Nov 03 23:23:01.844670 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.844626 4625 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7"} err="failed to get container status \"df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7\": rpc error: code = NotFound desc = could not find container \"df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7\": container with ID starting with df8202f4bccdae5dd754d9fc06a22b811debd4f85b6556265e2bbf638253b8a7 not found: ID does not exist" Nov 03 23:23:01.868129 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.867146 4625 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:23:01.868129 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.867180 4625 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a058ab5c-c053-4cca-aa99-f935d9292448-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:23:01.868129 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.867197 4625 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a058ab5c-c053-4cca-aa99-f935d9292448-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:23:01.868129 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.867212 4625 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbs2\" (UniqueName: \"kubernetes.io/projected/a058ab5c-c053-4cca-aa99-f935d9292448-kube-api-access-bjbs2\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\"" Nov 03 23:23:01.901793 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.893190 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5"] Nov 03 23:23:01.901793 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:01.897011 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-02e08d-7b78fffbc-49kz5"] Nov 03 23:23:02.185309 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:23:02.185273 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:23:02.727843 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:02.727811 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a058ab5c-c053-4cca-aa99-f935d9292448" path="/var/lib/kubelet/pods/a058ab5c-c053-4cca-aa99-f935d9292448/volumes" Nov 03 23:23:04.090976 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:04.090948 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:23:04.160618 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:04.160589 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64452pdx_26884d2c-6314-4cf8-b455-43dedd245166/extract/0.log" Nov 03 23:23:08.049039 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:23:08.049001 4625 scope.go:122] "RemoveContainer" containerID="3bab09998e51daed7517126b95a85883fbaa8de1c6dca32fe097298b2043e6df" Nov 03 23:23:32.186230 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:23:32.186186 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:24:02.186578 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:24:02.186534 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:24:08.082338 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:08.082298 4625 scope.go:122] "RemoveContainer" containerID="6e98e15850683afe9b715818ef20da2d60356104b320488caf11d3ff760c8b7d" Nov 03 23:24:16.517429 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:16.517341 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c9975f7bf-bm2l5_4471242b-9cd4-400f-bfc7-d1075e4c432d/manager/0.log" Nov 03 23:24:16.961294 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:16.961261 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7q6bd_c437aa1e-22e8-4e04-97d7-e8fcd06de290/manager/0.log" Nov 03 23:24:17.471015 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:17.470968 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9589f759-xg97m_91811bd7-2f89-4a68-8263-7d231eadc5b7/manager/0.log" Nov 03 23:24:18.837232 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:18.837199 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d897ffcf4-7fnds_8535d0ca-b280-4d43-abc1-2d9643eef8e8/maas-api/0.log" Nov 03 23:24:19.412407 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:19.412364 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-plmv7_f201a577-82e2-4218-8877-593ce43622d1/revoke-keys/0.log" Nov 03 23:24:19.606482 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:19.606449 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-8nstj_6cd3eb25-06aa-4f84-a19f-946149460fdf/revoke-keys/0.log" Nov 03 23:24:19.962292 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:19.962261 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b811c3ac-996zh_e29c0007-a41c-4906-b2cf-cfd8f9259991/revoke-keys/0.log" Nov 03 23:24:20.314619 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:20.314545 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rgzjd_de54298a-44f5-4c18-8800-cf79f2bab9c6/postgres/0.log" Nov 03 23:24:22.027254 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:22.027220 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cg9nj_7f061ce4-a7b7-4972-bd61-c812f53861a6/manager/0.log" Nov 03 23:24:22.233247 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:22.233204 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jxtx5_3917df96-a8b6-4304-8761-6d23a5f4ea2b/manager/0.log" Nov 03 23:24:22.653293 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:22.653263 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhpcd_c9c3eed8-d000-4186-a005-25d95469cb1f/manager/0.log" Nov 03 23:24:23.092646 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:23.092618 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xn8cx_7107b00f-9a1b-4b01-a596-b450fa81b1f4/manager/1.log" Nov 03 23:24:23.505639 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:23.505296 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xn8cx_7107b00f-9a1b-4b01-a596-b450fa81b1f4/manager/0.log" Nov 03 23:24:24.056786 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:24.056747 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9crbcbw_d899d6c6-8a9a-4f11-8063-5ece280dad35/istio-proxy/0.log" Nov 03 23:24:24.267116 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:24.267077 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vg6xv_6394ee67-e20e-4eb2-99a1-0c2aef09e7cd/discovery/0.log" Nov 03 23:24:25.428231 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:25.428193 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gprr2_e6eb71bf-3a09-4a93-b518-91961518a97e/istio-proxy/0.log" Nov 03 23:24:25.811825 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:25.811745 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-mfkdg_13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc/istio-proxy/0.log" Nov 03 23:24:26.600446 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:26.600420 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-zqt58_ef90202f-42a4-45ca-93b0-deebd7306fec/istio-proxy/0.log" Nov 03 23:24:26.792678 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:26.792647 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs_6c7f55fa-0e84-4ffc-9601-6c1c53c1b669/istio-proxy/0.log" Nov 03 23:24:28.360526 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:28.360492 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qt78b_2d8e5e86-4926-41f1-ba99-b2da2fab4889/main/0.log" Nov 03 23:24:28.760856 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:28.760753 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dwbc8_f072ea9a-f752-461e-85e1-3297b846fde0/main/0.log" Nov 03 23:24:29.133662 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:29.133631 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-7c5g6_6f8dc406-d4b0-407a-bd99-069aa88016cf/main/0.log" Nov 03 23:24:32.184958 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:24:32.184922 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:24:34.030238 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:34.030202 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj"] Nov 03 23:24:34.033256 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:34.033228 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8nstj"] Nov 03 23:24:34.727419 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:34.727385 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd3eb25-06aa-4f84-a19f-946149460fdf" path="/var/lib/kubelet/pods/6cd3eb25-06aa-4f84-a19f-946149460fdf/volumes" Nov 03 23:24:39.232839 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:39.232797 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c9975f7bf-bm2l5_4471242b-9cd4-400f-bfc7-d1075e4c432d/manager/0.log" Nov 03 23:24:39.688018 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:39.687983 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7q6bd_c437aa1e-22e8-4e04-97d7-e8fcd06de290/manager/0.log" Nov 03 23:24:40.142288 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:40.142249 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9589f759-xg97m_91811bd7-2f89-4a68-8263-7d231eadc5b7/manager/0.log" Nov 03 23:24:41.320028 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:41.319996 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d897ffcf4-7fnds_8535d0ca-b280-4d43-abc1-2d9643eef8e8/maas-api/0.log" Nov 03 23:24:41.936141 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:41.936108 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-plmv7_f201a577-82e2-4218-8877-593ce43622d1/revoke-keys/0.log" Nov 03 23:24:42.317048 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:42.316957 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b811c3ac-996zh_e29c0007-a41c-4906-b2cf-cfd8f9259991/revoke-keys/0.log" Nov 03 23:24:42.687259 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:42.687227 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rgzjd_de54298a-44f5-4c18-8800-cf79f2bab9c6/postgres/0.log" Nov 03 23:24:44.090547 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:44.090489 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cg9nj_7f061ce4-a7b7-4972-bd61-c812f53861a6/manager/0.log" Nov 03 23:24:44.289836 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:44.289802 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jxtx5_3917df96-a8b6-4304-8761-6d23a5f4ea2b/manager/0.log" Nov 03 23:24:44.682731 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:44.682680 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhpcd_c9c3eed8-d000-4186-a005-25d95469cb1f/manager/0.log" Nov 03 23:24:45.143072 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:45.143032 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xn8cx_7107b00f-9a1b-4b01-a596-b450fa81b1f4/manager/1.log" Nov 03 23:24:45.711605 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:45.711577 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9crbcbw_d899d6c6-8a9a-4f11-8063-5ece280dad35/istio-proxy/0.log" Nov 03 23:24:45.898984 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:45.898953 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vg6xv_6394ee67-e20e-4eb2-99a1-0c2aef09e7cd/discovery/0.log" Nov 03 23:24:47.050588 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:47.050558 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gprr2_e6eb71bf-3a09-4a93-b518-91961518a97e/istio-proxy/0.log" Nov 03 23:24:47.442528 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:47.442499 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-mfkdg_13369b2f-fa0e-42f6-8f6d-9d34c75ed1fc/istio-proxy/0.log" Nov 03 23:24:48.271292 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:48.271255 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-zqt58_ef90202f-42a4-45ca-93b0-deebd7306fec/istio-proxy/0.log" Nov 03 23:24:48.460523 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:48.460481 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-668d64f647-fg9gs_6c7f55fa-0e84-4ffc-9601-6c1c53c1b669/istio-proxy/0.log" Nov 03 23:24:49.718944 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:49.718909 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qt78b_2d8e5e86-4926-41f1-ba99-b2da2fab4889/main/0.log" Nov 03 23:24:50.094884 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:50.094844 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dwbc8_f072ea9a-f752-461e-85e1-3297b846fde0/main/0.log" Nov 03 23:24:50.450235 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:50.450118 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-7c5g6_6f8dc406-d4b0-407a-bd99-069aa88016cf/main/0.log" Nov 03 23:24:51.028656 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:51.028620 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7"] Nov 03 23:24:51.039254 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:51.039215 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plmv7"] Nov 03 23:24:52.726494 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:24:52.726458 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f201a577-82e2-4218-8877-593ce43622d1" path="/var/lib/kubelet/pods/f201a577-82e2-4218-8877-593ce43622d1/volumes" Nov 03 23:25:02.185580 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:25:02.185532 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:25:12.028482 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:25:12.028439 4625 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh"] Nov 03 23:25:12.039457 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:25:12.039425 4625 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b811c3ac-996zh"] Nov 03 23:25:12.727151 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:25:12.727116 4625 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29c0007-a41c-4906-b2cf-cfd8f9259991" path="/var/lib/kubelet/pods/e29c0007-a41c-4906-b2cf-cfd8f9259991/volumes" Nov 03 23:25:25.608826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:25:25.608784 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9589f759-xg97m_91811bd7-2f89-4a68-8263-7d231eadc5b7/manager/0.log" Nov 03 23:25:25.915097 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:25:25.915010 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9589f759-xg97m_91811bd7-2f89-4a68-8263-7d231eadc5b7/manager/0.log" Nov 03 23:25:32.184936 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:25:32.184894 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:26:02.186357 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:26:02.186316 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:26:08.128277 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:08.128242 4625 scope.go:122] "RemoveContainer" containerID="647549eb53b0f5dd6fa9aa9266cf497660f655c333e583de87d6d6b7f2fd0032" Nov 03 23:26:08.137348 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:08.137323 4625 scope.go:122] "RemoveContainer" containerID="c69d0a6d4f35a2324038787abe03f5a92b1352072cde0174ae3faae3abf00436" Nov 03 23:26:10.024142 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:10.024107 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9589f759-xg97m_91811bd7-2f89-4a68-8263-7d231eadc5b7/manager/0.log" Nov 03 23:26:10.328166 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:10.328065 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9589f759-xg97m_91811bd7-2f89-4a68-8263-7d231eadc5b7/manager/0.log" Nov 03 23:26:16.232702 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:16.232663 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nvbcj_17608ecb-fd95-49bf-a939-b2c1587e4fa1/global-pull-secret-syncer/0.log" Nov 03 23:26:17.510420 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:17.510383 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qwdnc_c2165f0f-049f-4588-898f-db7ef275411d/konnectivity-agent/0.log" Nov 03 23:26:17.612553 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:17.612515 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-81.ec2.internal_6cbffddcf197bf46e398ebd8b03a5cd7/haproxy/0.log" Nov 03 23:26:24.511004 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:24.510969 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cg9nj_7f061ce4-a7b7-4972-bd61-c812f53861a6/manager/0.log" Nov 03 23:26:24.636395 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:24.636363 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jxtx5_3917df96-a8b6-4304-8761-6d23a5f4ea2b/manager/0.log" Nov 03 23:26:24.943893 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:24.943860 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhpcd_c9c3eed8-d000-4186-a005-25d95469cb1f/manager/0.log" Nov 03 23:26:25.206046 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:25.205944 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xn8cx_7107b00f-9a1b-4b01-a596-b450fa81b1f4/manager/0.log" Nov 03 23:26:25.256881 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:25.256840 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xn8cx_7107b00f-9a1b-4b01-a596-b450fa81b1f4/manager/1.log" Nov 03 23:26:26.244952 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.244911 4625 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f"] Nov 03 23:26:26.245605 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.245580 4625 cpu_manager.go:414] "containerMap: removing container" podUID="a058ab5c-c053-4cca-aa99-f935d9292448" containerName="maas-api" Nov 03 23:26:26.245605 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.245603 4625 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a058ab5c-c053-4cca-aa99-f935d9292448" containerName="maas-api" Nov 03 23:26:26.245834 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.245741 4625 memory_manager.go:365] "RemoveStaleState removing state" podUID="a058ab5c-c053-4cca-aa99-f935d9292448" containerName="maas-api" Nov 03 23:26:26.249164 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.249143 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.275064 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.275031 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtpzb\"/\"openshift-service-ca.crt\"" Nov 03 23:26:26.275064 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.275049 4625 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qtpzb\"/\"default-dockercfg-8d4gm\"" Nov 03 23:26:26.275064 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.275054 4625 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtpzb\"/\"kube-root-ca.crt\"" Nov 03 23:26:26.278153 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.278121 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f"] Nov 03 23:26:26.342272 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.342225 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-proc\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.342467 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.342288 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-lib-modules\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.342467 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.342426 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-podres\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.342550 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.342537 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-sys\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.342603 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.342586 4625 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fzv\" (UniqueName: \"kubernetes.io/projected/24ffa064-adb6-47b2-b4aa-ae87b16926b9-kube-api-access-87fzv\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.443983 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-proc\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444022 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444038 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-lib-modules\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444068 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-podres\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444107 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-proc\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444122 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-sys\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444167 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-lib-modules\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444174 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-sys\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444183 4625 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fzv\" (UniqueName: \"kubernetes.io/projected/24ffa064-adb6-47b2-b4aa-ae87b16926b9-kube-api-access-87fzv\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.444313 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.444240 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24ffa064-adb6-47b2-b4aa-ae87b16926b9-podres\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.466700 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.466672 4625 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fzv\" (UniqueName: \"kubernetes.io/projected/24ffa064-adb6-47b2-b4aa-ae87b16926b9-kube-api-access-87fzv\") pod \"perf-node-gather-daemonset-5jn9f\" (UID: \"24ffa064-adb6-47b2-b4aa-ae87b16926b9\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.562826 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.562736 4625 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:26.870915 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:26.870879 4625 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f"] Nov 03 23:26:26.873940 ip-10-0-128-81 kubenswrapper[4625]: W1103 23:26:26.873907 4625 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod24ffa064_adb6_47b2_b4aa_ae87b16926b9.slice/crio-4d330677c6ebabf85210d427b26f82a3932b4bf27bad7f80a9c387b2d768a154 WatchSource:0}: Error finding container 4d330677c6ebabf85210d427b26f82a3932b4bf27bad7f80a9c387b2d768a154: Status 404 returned error can't find the container with id 4d330677c6ebabf85210d427b26f82a3932b4bf27bad7f80a9c387b2d768a154 Nov 03 23:26:27.069687 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:27.069627 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" event={"ID":"24ffa064-adb6-47b2-b4aa-ae87b16926b9","Type":"ContainerStarted","Data":"7ffcbe987e629185b92827600134bbc866c6443cf2eb0ccfbcc2a62c29077819"} Nov 03 23:26:27.069914 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:27.069705 4625 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" event={"ID":"24ffa064-adb6-47b2-b4aa-ae87b16926b9","Type":"ContainerStarted","Data":"4d330677c6ebabf85210d427b26f82a3932b4bf27bad7f80a9c387b2d768a154"} Nov 03 23:26:27.069914 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:27.069742 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:27.108889 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:27.108834 4625 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" podStartSLOduration=1.108819832 podStartE2EDuration="1.108819832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:27.106856238 +0000 UTC m=+4408.536192227" watchObservedRunningTime="2026-11-03 23:26:27.108819832 +0000 UTC m=+4408.538155825" Nov 03 23:26:27.644129 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:27.644091 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mxf8b_0ccd3e6d-5e97-4a53-b81e-8dde9c536497/kube-state-metrics/0.log" Nov 03 23:26:27.760060 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:27.760017 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mxf8b_0ccd3e6d-5e97-4a53-b81e-8dde9c536497/kube-rbac-proxy-main/0.log" Nov 03 23:26:27.882876 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:27.882837 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mxf8b_0ccd3e6d-5e97-4a53-b81e-8dde9c536497/kube-rbac-proxy-self/0.log" Nov 03 23:26:28.011351 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:28.011264 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dd945bbc8-ggpxf_9998718c-eddb-4aef-ac0d-2f15aa6b48ea/metrics-server/0.log" Nov 03 23:26:28.137408 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:28.137380 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-nb6fn_c3b4bc26-994e-4ad1-b961-83e96121136a/monitoring-plugin/0.log" Nov 03 23:26:28.644146 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:28.644113 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcbtf_6b8c9424-3148-40cb-ab34-c7cb86921f48/node-exporter/0.log" Nov 03 23:26:28.766704 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:28.766669 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcbtf_6b8c9424-3148-40cb-ab34-c7cb86921f48/kube-rbac-proxy/0.log" Nov 03 23:26:28.890191 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:28.890163 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcbtf_6b8c9424-3148-40cb-ab34-c7cb86921f48/init-textfile/0.log" Nov 03 23:26:29.385614 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:29.385587 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-45kb5_d0a56fa6-1997-4c34-8b30-fffd023081fb/kube-rbac-proxy-main/0.log" Nov 03 23:26:29.511636 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:29.511602 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-45kb5_d0a56fa6-1997-4c34-8b30-fffd023081fb/kube-rbac-proxy-self/0.log" Nov 03 23:26:29.638029 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:29.637940 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-45kb5_d0a56fa6-1997-4c34-8b30-fffd023081fb/openshift-state-metrics/0.log" Nov 03 23:26:31.367060 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:31.367027 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c59b76cb-ksvrz_5d2e3700-a62d-4a9b-a836-8771e39b0014/thanos-query/0.log" Nov 03 23:26:31.490001 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:31.489971 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c59b76cb-ksvrz_5d2e3700-a62d-4a9b-a836-8771e39b0014/kube-rbac-proxy-web/0.log" Nov 03 23:26:31.615424 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:31.615391 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c59b76cb-ksvrz_5d2e3700-a62d-4a9b-a836-8771e39b0014/kube-rbac-proxy/0.log" Nov 03 23:26:31.738977 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:31.738900 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c59b76cb-ksvrz_5d2e3700-a62d-4a9b-a836-8771e39b0014/prom-label-proxy/0.log" Nov 03 23:26:31.863640 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:31.863606 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c59b76cb-ksvrz_5d2e3700-a62d-4a9b-a836-8771e39b0014/kube-rbac-proxy-rules/0.log" Nov 03 23:26:31.993251 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:31.993166 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c59b76cb-ksvrz_5d2e3700-a62d-4a9b-a836-8771e39b0014/kube-rbac-proxy-metrics/0.log" Nov 03 23:26:32.185781 ip-10-0-128-81 kubenswrapper[4625]: E1103 23:26:32.185744 4625 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wkl88" containerName="extractor" Nov 03 23:26:33.086200 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:33.086173 4625 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-5jn9f" Nov 03 23:26:35.687349 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:35.687320 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cb57fd44f-zqgjf_95825889-a81f-4738-a105-1ff28b59e21f/console/0.log" Nov 03 23:26:39.273264 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:39.273223 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4hrjm_32c88bd7-b717-4197-b99c-4060840ef9d5/dns/0.log" Nov 03 23:26:39.394586 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:39.394557 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4hrjm_32c88bd7-b717-4197-b99c-4060840ef9d5/kube-rbac-proxy/0.log" Nov 03 23:26:39.541924 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:39.541835 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9crbcbw_d899d6c6-8a9a-4f11-8063-5ece280dad35/istio-proxy/0.log" Nov 03 23:26:40.166965 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:40.166923 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97blh_63ada7c0-b6ab-4ba2-bc6d-990b007311a1/dns-node-resolver/0.log" Nov 03 23:26:41.654032 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:41.653987 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g7nfl_e5eb73d7-8dd5-4e38-b312-9fd9f4881762/node-ca/0.log" Nov 03 23:26:43.788529 ip-10-0-128-81 kubenswrapper[4625]: I1103 23:26:43.788473 4625 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9crbcbw_d899d6c6-8a9a-4f11-8063-5ece280dad35/istio-proxy/0.log"